Skip to content

fixes for cbs process files to support importing old data #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fixes for cbs process files to support importing old data #16

wants to merge 3 commits into from

Conversation

@OriHoch
Copy link
Contributor

@OriHoch OriHoch commented Nov 21, 2021

#9

@OriHoch OriHoch self-assigned this Nov 21, 2021
@OriHoch OriHoch marked this pull request as draft November 21, 2021 08:26
@OriHoch
Copy link
Contributor Author

OriHoch commented Nov 21, 2021

manually deployed to anyway-dev for testing

@OriHoch
Copy link
Contributor Author

OriHoch commented Nov 21, 2021

dag is running.. https://dev-airflow.anyway.co.il/graph?root=&dag_id=cbs&base_date=2021-11-21T08%3A35%3A41Z&num_runs=25&execution_date=2021-11-21T08%3A35%3A40.282541%2B00%3A00&arrange=LR

@OriHoch
Copy link
Contributor Author

OriHoch commented Nov 21, 2021

looks like there is a problem with the data, check-data-in-datastore step failed - https://dev-airflow.anyway.co.il/log?dag_id=cbs&task_id=check-data-in-datastore&execution_date=2021-11-21T08%3A35%3A40.282541%2B00%3A00

@OriHoch OriHoch changed the title remove the rows limit on process_files to allow importing historical data fixes for cbs process files to support importing old data Dec 27, 2021
@OriHoch
Copy link
Contributor Author

OriHoch commented Dec 27, 2021

running on dev: https://dev-airflow.anyway.co.il/graph?root=&dag_id=cbs&base_date=2021-12-27T08%3A22%3A18Z&num_runs=25&execution_date=2021-12-27T08%3A22%3A17.886570%2B00%3A00&arrange=LR

@OriHoch
Copy link
Contributor Author

OriHoch commented Dec 27, 2021

run completed successfully, looks like it works

@OriHoch OriHoch marked this pull request as ready for review December 27, 2021 10:13
@OriHoch
Copy link
Contributor Author

OriHoch commented Dec 27, 2021

assigning to @atalyaalon to merge and deploy (I'm not sure how it might affect existing data on production, and what kind of preparation / testing should be done)

@OriHoch OriHoch assigned atalyaalon and unassigned OriHoch Dec 27, 2021
@OriHoch OriHoch mentioned this pull request Dec 27, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@carmelp16 carmelp16 Awaiting requested review from carmelp16

@atalyaalon atalyaalon Awaiting requested review from atalyaalon

At least 1 approving review is required to merge this pull request.

Assignees

@atalyaalon atalyaalon

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@OriHoch @atalyaalon