Airflow process - CBS data backfill

[atalyaalon]

Create an airflow process that allows CBS data backfill from s3 (without importing from email) - and with a load_start_year parameter that can be changed by the airflow user The relevant command: (python main.py process cbs --source s3 --load_start_year 2020
We had a Jenkins process that enabled such backfill.

[OriHoch]

we are not using S3 at the moment, we have all the files in the airflow data directory - https://airflow-data.anyway.co.il/cbs/files/

[OriHoch]

there is no point to have load_start_year 2020 as that is the default (current year - 1)

[OriHoch]

development complete, deployed to dev
to run a back-fill - do a manual dag run with the following example json:

{"load_start_year": 2019}

initiated 2 dag runs on dev for testing:

1. default run without any parameters
2. run with load_start_year 2019

assigning to @atalyaalon to test and make a release

[atalyaalon]

@OriHoch seems like 2020 data is not in our DB even though default load_start_year is 2020 as you've mentioned - seems like data from 2020 and 2021 in that case is deleted however only data from 2021 is loaded in the CBS process (from s3). Can you take a look?

[OriHoch]

how do you check this?

[atalyaalon]

@OriHoch
When load start year is 2020, then data starting 2020 and on is deleted.
Now the question is what data is loaded from s3? I assume datat from 2020 is not loaded for some reason

[OriHoch]

we are not loading any data from s3

how did you check that data in DB from 2020 is not loaded?