Help with CDC setup using Airbyte Connectors #59684
Unanswered
nikhilism15
asked this question in
Platform General Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am really interested in how the airbyte system works, and I wanted to setup a small POC to perform CDC using a command line program.
I am setting up two connectors separately (one source - MSSQL, one destination - PostgresQL). I have setup the source database properly for CDC. When I run the read operation, I am getting the initial data properly, and I am getting some state object as well, which I store and pass to the next operation.
Now I perform some inserts/updates/deletes and run the read + write operation again (this time with the state.json parameter included to the read operation). Although the inserts and updates are copied properly (maybe because in the catalog.json I set the sync_mode and destination_sync_mode to Incremental + Append Dedup), the delete operation does not seem to be propagated to the target table - the target table still has the (supposedly) deleted record, although with the older sync_id. I am not sure exactly what's wrong with my setup, as other changes are propagating fine.
I did setup abctl in my local system and setup the same table for CDC and deletes are being reflected in the Target.
Could someone shed some light on what's missing? I will be happy to provide more detail if required.
Beta Was this translation helpful? Give feedback.
All reactions