Skip to content
This repository was archived by the owner on Sep 16, 2025. It is now read-only.
This repository was archived by the owner on Sep 16, 2025. It is now read-only.

Handling conflicts and errors for dataset transfers #27

@sunank200

Description

@sunank200

Please describe the feature you'd like to see
There are multiple scenarios to handle dataset transfers:

  1. Lets take a scenario where we are transferring 100 rows from snowflake to bigquery and error occurs after transferring the 99th row. What's the correct way to handle this?
    • By default should this be a TRANSACTION and roll back all the changes done?
    • Or should it throw the error for all the rows that error out? What's the right way to error out for a larger number of rows that has the error

This should be handled before 1.0 release. To start with we can replicate something similar to Fivetran does.

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions