This repository was archived by the owner on May 22, 2025. It is now read-only.
Support partitions when tables are created and be able to replace a partition #1865
jherrmannNetfonds
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi everybody,
When ingesting data into a cloud data warehouse (BigQuery, Snowflake etc.) I want to partition my data (for example by execution_date), so that when I clear/rerun a DAG, only the data of the affected partition is replaced.
Currently Tables cannot automatically created with a partition. Maybe the
metadata
parameter of theTable
Class would be a option to add something like that. When I use the append operator I should be able to replace a partition.Are there any ideas into this direction yet?
I am not sure, what would be a good place to implement a "replacement of a partition", since the sdk has several operators which can store data into a cloud DWH, but with different option on how to store the data in the target table. For example has the load_file operator has more option to change the load behavior then a transform operator. At least as far as I understand.
Beta Was this translation helpful? Give feedback.
All reactions