Weekly Processing?

[davies-w]

I'd like to be able to run a dvc pipeline that will maintain a weekly state..

EG:

Input is:
s3:week-1.df.gz
s3:week-2.df.gz

Intermediate Output:
dvc_data/intermediate/week-1-processed.df
dvc_data/intermediate/week-2-processed.df

Final Output:
dvc_data/final/combined-formatted.dataset

So that when a new s3:week-3.df.gz appears, dvc will just run on that file, and produce:

dvc_data/intermediate/week-3-processed.df

and then updates the weeks together to produce:

dvc_data/final/combined-formatted.dataset

Extra credit if you can suck in the original version of dvc_data/final/combined-formatted.dataset and merge it with dvc_data/intermediate/week-3-processed.df

[shcheklein]

Hi @davies-w . A few options:

• Use persist flag for output(s). Find it in this doc
• Use DataChain + DVC (and in DataChain you can describe diff between datasets on S3, etc). Check this example to get started: https://github.com/shcheklein/example-datachain-dvc . Let me know if you need more details / expand on that.

[davies-w]

Thanks so much indeed! Will follow both these leads!