Skip to content

fix: add sql transaction by log when importing #925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 19, 2025
Merged

fix: add sql transaction by log when importing #925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deployments/pulumi/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.23.1
toolchain go1.23.3

require (
github.com/formancehq/go-libs/v3 v3.0.0-20250407102820-68fa020e67bb
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539
github.com/google/uuid v1.6.0
github.com/invopop/jsonschema v0.13.0
github.com/kos-v/dsnparser v1.1.0
Expand Down
4 changes: 2 additions & 2 deletions deployments/pulumi/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,8 @@ github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/formancehq/go-libs/v3 v3.0.0-20250407102820-68fa020e67bb h1:bX/OQdZJryJlOzc1Ao8mMKg1kV9qSmmJ1aHxv5N71HE=
github.com/formancehq/go-libs/v3 v3.0.0-20250407102820-68fa020e67bb/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539 h1:6kUkmD2GiZGB7TDpGaPas2ipaAKqP/os3PVk4XFVrpI=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/gliderlabs/ssh v0.3.8 h1:a4YXD1V7xMF9g5nTkdfnja3Sxy1PVDCj1Zg4Wb8vY6c=
github.com/gliderlabs/ssh v0.3.8/go.mod h1:xYoytBv1sV0aL3CavoDuJIQNURXkkfPA/wxQ1pL1fAU=
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI=
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ require (
)

require (
github.com/formancehq/go-libs/v3 v3.0.0-20250408113106-7b3525e0b25b
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539
github.com/robfig/cron/v3 v3.0.1
github.com/spf13/viper v1.20.1
gopkg.in/yaml.v3 v3.0.1
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/formancehq/go-libs/v3 v3.0.0-20250408113106-7b3525e0b25b h1:EvnYMGyZo+fq9fsrraedKBlVTbeE2mqpJ/edj8FRTtU=
github.com/formancehq/go-libs/v3 v3.0.0-20250408113106-7b3525e0b25b/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539 h1:6kUkmD2GiZGB7TDpGaPas2ipaAKqP/os3PVk4XFVrpI=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/formancehq/numscript v0.0.16 h1:kNNpPTmTvhRUrMXonZPMwUXUpJ06Io1WwC56Yf3nr1E=
github.com/formancehq/numscript v0.0.16/go.mod h1:8WhBIqcK6zu27njxy7ZG7CaDX0MHtI9qF9Ggfj07wfU=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
Expand Down
51 changes: 35 additions & 16 deletions internal/controller/ledger/controller_default.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,21 +204,40 @@
}
lastLogID = log.ID

if err := ctrl.importLog(ctx, log); err != nil {
switch {
case errors.Is(err, postgres.ErrSerialization) ||
errors.Is(err, ErrConcurrentTransaction{}):
return NewErrImport(errors.New("concurrent transaction occur" +
"red, cannot import the ledger"))
store, _, err := ctrl.store.BeginTX(ctx, nil)
if err != nil {
return fmt.Errorf("starting transaction: %w", err)
}

Check warning on line 210 in internal/controller/ledger/controller_default.go

View check run for this annotation

Codecov / codecov/patch

internal/controller/ledger/controller_default.go#L209-L210 

Added lines #L209 - L210 were not covered by tests
err = func() error {
defer func() {
_ = store.Rollback()
}()

if err := ctrl.importLog(ctx, store, log); err != nil {
switch {

Check warning on line 217 in internal/controller/ledger/controller_default.go

View check run for this annotation

Codecov / codecov/patch

internal/controller/ledger/controller_default.go#L217 

Added line #L217 was not covered by tests
case errors.Is(err, postgres.ErrSerialization) ||
errors.Is(err, ErrConcurrentTransaction{}):
return NewErrImport(errors.New("concurrent transaction occur" +
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a quick one -- shouldn't they all be "NewErrImport" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm used to define specific errors for business errors.
In the context of the ledger, errors are processed on the api by checking for each kind of error (a bit like a try/catch).
All unknown errors return a 500 and trigger an error log (and trace). Others are translated to a specific business error code.
If we wrap all errors in NewErrImport, technical errors could leak in the api.
I guess we could have two levels of checks api side.

"red, cannot import the ledger"))

Check warning on line 221 in internal/controller/ledger/controller_default.go

View check run for this annotation

Codecov / codecov/patch

internal/controller/ledger/controller_default.go#L219-L221 

Added lines #L219 - L221 were not covered by tests
}
return fmt.Errorf("importing log %d: %w", *log.ID, err)

Check warning on line 223 in internal/controller/ledger/controller_default.go

View check run for this annotation

Codecov / codecov/patch

internal/controller/ledger/controller_default.go#L223 

Added line #L223 was not covered by tests
}

if err := store.Commit(); err != nil {
return fmt.Errorf("committing transaction: %w", err)

Check warning on line 227 in internal/controller/ledger/controller_default.go

View check run for this annotation

Codecov / codecov/patch

internal/controller/ledger/controller_default.go#L227 

Added line #L227 was not covered by tests
}
return fmt.Errorf("importing log %d: %w", *log.ID, err)

return nil
}()
if err != nil {
return err

Check warning on line 233 in internal/controller/ledger/controller_default.go

View check run for this annotation

Codecov / codecov/patch

internal/controller/ledger/controller_default.go#L233 

Added line #L233 was not covered by tests
}
}

return err
}

func (ctrl *DefaultController) importLog(ctx context.Context, log ledger.Log) error {
func (ctrl *DefaultController) importLog(ctx context.Context, store Store, log ledger.Log) error {
_, err := tracing.Trace(
ctx,
ctrl.tracer,
Expand All @@ -227,33 +246,33 @@
switch payload := log.Data.(type) {
case ledger.CreatedTransaction:
logging.FromContext(ctx).Debugf("Importing transaction %d", *payload.Transaction.ID)
if err := ctrl.store.CommitTransaction(ctx, &payload.Transaction, payload.AccountMetadata); err != nil {
if err := store.CommitTransaction(ctx, &payload.Transaction, payload.AccountMetadata); err != nil {
return nil, fmt.Errorf("failed to commit transaction: %w", err)
}
logging.FromContext(ctx).Debugf("Imported transaction %d", *payload.Transaction.ID)
case ledger.RevertedTransaction:
logging.FromContext(ctx).Debugf("Reverting transaction %d", *payload.RevertedTransaction.ID)
_, _, err := ctrl.store.RevertTransaction(
_, _, err := store.RevertTransaction(
ctx,
*payload.RevertedTransaction.ID,
*payload.RevertedTransaction.RevertedAt,
)
if err != nil {
return nil, fmt.Errorf("failed to revert transaction: %w", err)
}
if err := ctrl.store.CommitTransaction(ctx, &payload.RevertTransaction, nil); err != nil {
if err := store.CommitTransaction(ctx, &payload.RevertTransaction, nil); err != nil {
return nil, fmt.Errorf("failed to commit transaction: %w", err)
}
case ledger.SavedMetadata:
switch payload.TargetType {
case ledger.MetaTargetTypeTransaction:
logging.FromContext(ctx).Debugf("Saving metadata of transaction %d", payload.TargetID)
if _, _, err := ctrl.store.UpdateTransactionMetadata(ctx, payload.TargetID.(uint64), payload.Metadata, log.Date); err != nil {
if _, _, err := store.UpdateTransactionMetadata(ctx, payload.TargetID.(uint64), payload.Metadata, log.Date); err != nil {
return nil, fmt.Errorf("failed to update transaction metadata: %w", err)
}
case ledger.MetaTargetTypeAccount:
logging.FromContext(ctx).Debugf("Saving metadata of account %s", payload.TargetID)
if err := ctrl.store.UpdateAccountsMetadata(ctx, ledger.AccountMetadata{
if err := store.UpdateAccountsMetadata(ctx, ledger.AccountMetadata{
payload.TargetID.(string): payload.Metadata,
}); err != nil {
return nil, fmt.Errorf("failed to update account metadata: %w", err)
Expand All @@ -263,20 +282,20 @@
switch payload.TargetType {
case ledger.MetaTargetTypeTransaction:
logging.FromContext(ctx).Debugf("Deleting metadata of transaction %d", payload.TargetID)
if _, _, err := ctrl.store.DeleteTransactionMetadata(ctx, payload.TargetID.(uint64), payload.Key, log.Date); err != nil {
if _, _, err := store.DeleteTransactionMetadata(ctx, payload.TargetID.(uint64), payload.Key, log.Date); err != nil {
return nil, fmt.Errorf("failed to delete transaction metadata: %w", err)
}
case ledger.MetaTargetTypeAccount:
logging.FromContext(ctx).Debugf("Deleting metadata of account %s", payload.TargetID)
if err := ctrl.store.DeleteAccountMetadata(ctx, payload.TargetID.(string), payload.Key); err != nil {
if err := store.DeleteAccountMetadata(ctx, payload.TargetID.(string), payload.Key); err != nil {
return nil, fmt.Errorf("failed to delete account metadata: %w", err)
}
}
}

logCopy := log
logging.FromContext(ctx).Debugf("Inserting log %d", *log.ID)
if err := ctrl.store.InsertLog(ctx, &log); err != nil {
if err := store.InsertLog(ctx, &log); err != nil {
return nil, fmt.Errorf("failed to insert log: %w", err)
}

Expand Down
2 changes: 1 addition & 1 deletion test/rolling-upgrades/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ require (
github.com/cyphar/filepath-securejoin v0.3.6 // indirect
github.com/djherbis/times v1.5.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/formancehq/go-libs/v3 v3.0.0-20250407102820-68fa020e67bb // indirect
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.1 // indirect
Expand Down
4 changes: 2 additions & 2 deletions test/rolling-upgrades/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ github.com/emirpasic/gods v1.18.1/go.mod h1:8tpGGwCnJ5H4r6BWwaV6OrWmMoPhUl5jm/FM
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/formancehq/go-libs/v3 v3.0.0-20250407102820-68fa020e67bb h1:bX/OQdZJryJlOzc1Ao8mMKg1kV9qSmmJ1aHxv5N71HE=
github.com/formancehq/go-libs/v3 v3.0.0-20250407102820-68fa020e67bb/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539 h1:6kUkmD2GiZGB7TDpGaPas2ipaAKqP/os3PVk4XFVrpI=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/gliderlabs/ssh v0.3.8 h1:a4YXD1V7xMF9g5nTkdfnja3Sxy1PVDCj1Zg4Wb8vY6c=
Expand Down
2 changes: 1 addition & 1 deletion tools/generator/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ replace github.com/formancehq/ledger => ../..
replace github.com/formancehq/ledger/pkg/client => ../../pkg/client

require (
github.com/formancehq/go-libs/v3 v3.0.0-20250408113106-7b3525e0b25b
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539
github.com/formancehq/ledger v0.0.0-00010101000000-000000000000
github.com/formancehq/ledger/pkg/client v0.0.0-00010101000000-000000000000
github.com/spf13/cobra v1.9.1
Expand Down
4 changes: 2 additions & 2 deletions tools/generator/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/formancehq/go-libs/v3 v3.0.0-20250408113106-7b3525e0b25b h1:EvnYMGyZo+fq9fsrraedKBlVTbeE2mqpJ/edj8FRTtU=
github.com/formancehq/go-libs/v3 v3.0.0-20250408113106-7b3525e0b25b/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539 h1:6kUkmD2GiZGB7TDpGaPas2ipaAKqP/os3PVk4XFVrpI=
github.com/formancehq/go-libs/v3 v3.0.0-20250422113236-ec98813a1539/go.mod h1:mRr5/y0I64iJ4I+BXNkUy49izwrh3SA5L+MTWD1d/7Q=
github.com/formancehq/numscript v0.0.16 h1:kNNpPTmTvhRUrMXonZPMwUXUpJ06Io1WwC56Yf3nr1E=
github.com/formancehq/numscript v0.0.16/go.mod h1:8WhBIqcK6zu27njxy7ZG7CaDX0MHtI9qF9Ggfj07wfU=
github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
Expand Down
Loading