source-postgres running ctid then xmin on initial load duplicating cache records #60999
Unanswered
msharpe1
asked this question in
Connector Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I am using Pyairbyte (0.24.2) with source-postgres connector (3.2.24 but also used latest) and experiencing behaviour below.
Setup
Source DB (Postgres) -> Pyairbyte -> Postgres backed Cache
'replication_method': {'method': 'Xmin'}
Mode: Full refresh and also tried incremental with the same result
Result
The end result is a duplication of the same record within the cache as both ctid and xmin are executed.
Is this intended behaviour?
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.PostgresSource(getIncrementalIterators):536 Streams to be synced via ctid : 1
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.PostgresSource(getIncrementalIterators):537 Streams: audit.tabletest
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.PostgresSource(getIncrementalIterators):545 Streams to be synced via xmin : 1
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.PostgresSource(getIncrementalIterators):546 Streams: audit.tabletest
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.c.PostgresCtidHandler(queryTableCtid):157 Queueing query for table: tabletest
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.x.PostgresXminHandler(queryTableXmin):128 Queueing query for table: tabletest
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.c.InitialSyncCtidIterator(ctidQueryPlan):220 Will read 131072 pages to get 1GB
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.PostgresQueryUtils(fileNodeForIndividualStream):235 Relation filenode is for stream "audit"."tabletest" is 275515
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.c.InitialSyncCtidIterator(createCtidQueryStatement):277 Preparing query for table: tabletest
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.c.InitialSyncCtidIterator(createCtidQueryStatement):292 Executing query for table tabletest: SELECT ctid::text, "column_1","column_2" FROM "audit"."tabletest" WHERE ctid > ?::tid with binding (0,0)
2025-05-29 19:29:28 - INFO - INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(initialize):24 Set initial fetch size: 10 rows
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.PostgresQueryUtils(fileNodeForIndividualStream):235 Relation filenode is for stream "audit"."tabletest" is 275515
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.c.InitialSyncCtidIterator(computeNext):118 The latest file node 275515 for stream audit_tabletest is equal to the last file node 275515 known to Airbyte.
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.c.CtidStateManager(createFinalStateMessage):115 Finished initial sync of stream audit_tabletest, Emitting final state, state is io.airbyte.protocol.models.v0.AirbyteStateMessage@736ac09a[type=STREAM,stream=io.airbyte.protocol.models.v0.AirbyteStreamState@6ecd665[streamDescriptor=io.airbyte.protocol.models.v0.StreamDescriptor@45394b31[name=tabletest,namespace=audit,additionalProperties={}],streamState={"version":2,"state_type":"xmin","num_wraparound":0,"xmin_xid_value":430264,"xmin_raw_value":430264},additionalProperties={}],global=,data=,sourceStats=,destinationStats=,additionalProperties={}]
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.x.PostgresXminHandler(createXminQueryStatement):150 Preparing query for table: tabletest
2025-05-29 19:29:28 - INFO - INFO main i.a.i.s.p.x.PostgresXminHandler(createXminQueryStatement):160 Executing query for table tabletest: HikariProxyPreparedStatement@2082678778 wrapping SELECT "column_1","column_2" FROM "audit"."tabletest" WHERE xmin::text::bigint >= ('0'::int8)
2025-05-29 19:29:28 - INFO - INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(initialize):24 Set initial fetch size: 10 rows
Beta Was this translation helpful? Give feedback.
All reactions