How can I partial partitioned Asset Dependencies #31666
Unanswered
william-pearsall-foodsmart
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
We have a partitioned asset using MultiPartition to partition on "data_source" and "date", we'll call this Asset B. We also need to build an upstream asset for this that does some preprocessing Asset A. Asset B should depend on Asset A, but only for a specific subset of data_source partitions. Not all of the configured partitions will require this preprocessing step. We want this to be assets as the preprocessing has to do with different source file processing and some data_source partitions will be delivering extra files and others won't.
I'm not certain how we can implement this in the current dependency system from the examples I've seen.
Beta Was this translation helpful? Give feedback.
All reactions