Destination BigQuery: add the ability to partition by _airbyte_extracted_at #53198
KimPlv
started this conversation in
Connector Ideas and Features
Replies: 1 comment
-
I think this is a great idea. I have a very large source which never requires a refresh of data (hence _airbyte_generation_id never increments) and I end up having huge compute costs since it is not partitioned on _airbyte_extracted_at. Partitioning on extracted_at would be 2 orders of magnitude more efficient in the jobs bytes processed in my case |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
Is it possible to give the option of selecting the partition field between _airbyte_generation_id and _airbyte_extracted_at for Bigquery tables. It seems more optimized to reverse and have _airbyte_extracted_at for the partition and _airbyte_generation_id for clustering. (Btw why did you choose this configuration? ) But at least, if we can have the possibility of having the old version of the partition as an option, that would be great.
Beta Was this translation helpful? Give feedback.
All reactions