Sweets only checks for presence of files not of files needed

[whatnick]

Sweets only checks if some files exist before skipping a step. If sweets config is changed to add more temporal depth the extra files are not reprocessed.

It would be good to :

1. Perform a deep check to see if the needed files exist
2. Run a reset command on a config to clean up all outputs and preserve .SAFE files downloaded since that is slow to redo.

I have been doing step 2 using git clean -df since I am running from the git checkout folder.

[scottstanie]

Have you seen the skip_download_if_exists option in the config? Did setting that to false not work?

[whatnick]

It is set to true, but in reality we need x3 options - true, false or append.

[whatnick]

Also not validating the files leads to errors such as this. The aria2 checksum approach seems deprecated.

[08/14/25 17:58:48] INFO     Setting up 4 workers for ThreadPoolExecutor                                                                                                                                                  core.py:544
[08/14/25 17:58:48] INFO     Using cached burst DB at /home/tisham/.cache/sweets/burst_map_bbox_only.sqlite3                                                                                                          _burst_db.py:44
[08/14/25 17:58:48] INFO     DEM already exists: /mnt/c/Users/tisha/dev/sweets/dem.tif                                                                                                                                      dem.py:22
[08/14/25 17:58:48] INFO     Total elapsed time for sweets.dem.create_dem : 0.00 minutes (0.01 seconds)                                                                                                                   _log.py:111
                    INFO     Water mask already exists: /mnt/c/Users/tisha/dev/sweets/watermask.flg                                                                                                                         dem.py:43
                    INFO     Total elapsed time for sweets.dem.create_water_mask : 0.00 minutes (0.01 seconds)                                                                                                            _log.py:111
                    INFO     Found 12 existing files in /mnt/c/sweets/data. Skipping download.                                                                                                                            core.py:324
                    INFO     Using cached burst DB at /home/tisham/.cache/sweets/burst_map_bbox_only.sqlite3                                                                                                          _burst_db.py:44
[08/14/25 17:58:48] INFO     Orbit search_path: /mnt/c/sweets/data, save_dir: /mnt/c/Users/tisha/dev/sweets/orbits                                                                                                       _orbit.py:13
journal: Generating burst map for 12 SAFE files
Traceback (most recent call last):
  File "/home/tisham/micromamba/envs/sweets-env/bin/sweets", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/sweets/cli.py", line 237, in main
    func(arg_dict)
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/sweets/cli.py", line 217, in run_workflow
    workflow.run(starting_step=kwargs.get("starting_step", 1))
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/sweets/_log.py", line 101, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/sweets/core.py", line 566, in run
    self._geocode_slcs(rslc_files, self.dem_filename, burst_db_file)
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/sweets/_log.py", line 101, in wrapper
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/sweets/core.py", line 341, in _geocode_slcs
    compass_cfg_files = create_config_files(
                        ^^^^^^^^^^^^^^^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/sweets/_geocode_slcs.py", line 159, in create_config_files
    s1_geocode_stack.run(
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/compass/s1_geocode_stack.py", line 429, in run
    burst_map = generate_burst_map(
                ^^^^^^^^^^^^^^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/compass/s1_geocode_stack.py", line 115, in generate_burst_map
    ref_bursts = load_bursts(zip_file, orbit_path, subswath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/s1reader/s1_reader.py", line 1270, in load_bursts
    bursts = _burst_from_zip(path, id_str, orbit_path, flag_apply_eap)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/site-packages/s1reader/s1_reader.py", line 1313, in _burst_from_zip
    with zipfile.ZipFile(zip_path, "r") as z_file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/zipfile/__init__.py", line 1354, in __init__
    self._RealGetContents()
  File "/home/tisham/micromamba/envs/sweets-env/lib/python3.12/zipfile/__init__.py", line 1421, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

[scottstanie]

That second error seems like a separate issue. Did a file get corrupted? I don't know how sweets would discern this other than how you are seeing either that error.

As for append mode, I don't know if I'm understanding the request, as there are many possible expectations for what a user might want for "auto update". That is why I stayed away from building it so far- it could mean something very complicated.
What were you picturing for that?

[whatnick]

The download was interrupted. In the broader context we are looking to run Sweets as a pod in kubernetes controlled by Argo workflows to process large areas of Australia, much the way OPERA products are created. Interruption of the download process is expected as pods get deleted. Just working out the sharp edges of Sweets and where we need to engineer in additional retries / checks outside of your excellent workflow.