Add different download directories for JRA55 and Bathymetry

[simone-silvestri]

I have noticed that JRA55 and Bathymetry tests might struggle because they are split into CPU and GPU tests. Sometimes, one of the tests downloads the files while the other wants to read them before the downloading process is complete, and the tests fail with a NetCDF (101) error. By splitting the download directory into two different paths for CPU and GPU, this problem can be solved.

This PR also introduces a download directory for the ECCO module directly in the ECCOMetadata type. This is a first step to generalize the Metadata type for all the data we need to download

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 16 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (fc175f6) to head (5f403a9).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/DataWrangling/ECCO/ECCO_metadata.jl	0.00%	8 Missing ⚠️
src/DataWrangling/ECCO/ECCO.jl	0.00%	6 Missing ⚠️
src/DataWrangling/ECCO/ECCO_mask.jl	0.00%	1 Missing ⚠️
src/DataWrangling/ECCO/ECCO_restoring.jl	0.00%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #180   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         31      31           
  Lines       1730    1732    +2     
=====================================
- Misses      1730    1732    +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[glwagner]

what? why?

[simone-silvestri]

I could also revert it back, but it is nice to know the path of the directory where the data resides, compactly in the metadata information

[glwagner]

The PR isn't well motivated.

The problem described has to do with improper initialization of the tests. During test initialization we apparently should download all relevant data. This can be done by adding to the init test group here;

ClimaOcean.jl/test/runtests.jl

Lines 7 to 10 in ddea939

	if test_group == :init || test_group == :all
	using CUDA
	CUDA.precompile_runtime()
	end

I'm not sure about the other changes regarding download directories. Maybe are positive. But making these changes for tests is clearly wrong.

[glwagner]

This PR won't fix the issue that will arise eventually (hopefully) when more than one test group has to use the downloaded bathymetry data.

[simone-silvestri]

Ok i wll try downloading data in the init step

[simone-silvestri]

This PR should be ready to go. We have to remember to add the data we download in the tests in the init when adding new tests.

[glwagner]

I think init for any module should always go in the top-level file. It's going to be hard to find otherwise. Plus we want the top-level file to give an overview of the module.

[gaelforget]

Was trying to run examples/near_global_ocean_simulation.jl but am getting the error below. Would this branch help?

ps. NOAA servers struggled during past weeks after Hurricane Helene, which may explain the 503 error

[ Info:  ... downloaded 412.0 B (100% complete, 33.845 seconds)
--2024-10-12 13:54:38--  https://www.ngdc.noaa.gov/thredds/fileServer/global/ETOPO2022/60s/60s_surface_elev_netcdf/ETOPO_2022_v1_60s_N90W180_surface.nc
Resolving www.ngdc.noaa.gov (www.ngdc.noaa.gov)... 140.172.190.1
Connecting to www.ngdc.noaa.gov (www.ngdc.noaa.gov)|140.172.190.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable

[glwagner]

Was trying to run examples/near_global_ocean_simulation.jl but am getting the error below. Would this branch help?

ps. NOAA servers struggled during past weeks after Hurricane Helene, which may explain the 503 error

[ Info:  ... downloaded 412.0 B (100% complete, 33.845 seconds)
--2024-10-12 13:54:38--  https://www.ngdc.noaa.gov/thredds/fileServer/global/ETOPO2022/60s/60s_surface_elev_netcdf/ETOPO_2022_v1_60s_N90W180_surface.nc
Resolving www.ngdc.noaa.gov (www.ngdc.noaa.gov)... 140.172.190.1
Connecting to www.ngdc.noaa.gov (www.ngdc.noaa.gov)|140.172.190.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable

Huh, this looks like an issue downloading the etopo data (which happens in regrid_bathymetry:

ClimaOcean.jl/examples/near_global_ocean_simulation.jl

Lines 55 to 58 in fc175f6

	bottom_height = regrid_bathymetry(grid;
	minimum_depth = 10,
	interpolation_passes = 5,
	major_basins = 3)

@gaelforget can you check to see if it is still broken?

Another solution is you can provide a filename and dir to regrid_bathymetry which should avoid the download. I am not sure this is tested yet.