CI tests take 20 mins

[peterdudfield]

Is there a way to speed up CI tests?
Locally they are a lot quicker for me?
Investigate what is taking a long time in CI

[laurenceandrews]

Happy to take a look at this as my first issue here! Let me know your thoughts/any info I'm missing on the current CI setup as this is my very first glance at the code.

A full pytest runs in around 3:45 for me locally. Doesn't seem like I have permission to re-run somebody else's Github action to test CI speed. I'm assuming if I make my own branch and PR, this would be the easiest way to test CI speed?

---

CI runners suffer from several things (other than generally limited resources):

1. Cold environment

• Dependencies, environments, and temp files are re-installed and rebuilt every run (this is probably the main difference maker if it's not something we're already conscious of in the CI setup).

2. Higher network latency

• CI runners can experience slower external API calls or downloads compared to a fast, stable local network.

3. Restrictions on parallelisation

• Tests that run in parallel locally (if they do at all) may not be optimally configured to do so in CI.

4. Slower disk I/O

• File reading/writing and large dataset/temp file creation is often slower on CI runners compared to local SSDs.

---

Below are some areas we could look at to help tackle the above:

1. Cold Environment

• Cache Python dependencies using GitHub Actions cache to avoid reinstalling them every run.
• Explore caching temporary or intermediate files created during tests.

2. Higher Network Latency

• Replace API or external service calls with mocks where possible to eliminate network delays.
• Consider using pre-downloaded static datasets or artifacts in place of live network fetches.

3. Restrictions on Parallelisation

• Look into pytest-xdist to run tests in parallel within the CI environment.
• Adjust parallelization settings in pytest to find the optimal number of workers for CI’s limited resources.

4. Slower Disk I/O

• Reduce reliance on larger temporary datasets or files during tests, or mock their creation.
• Optimize file operations by using memory-efficient formats or in-memory operations (e.g., StringIO instead of actual files).
• Profile specific tests that rely heavily on I/O to identify bottlenecks and optimize them.

[peterdudfield]

Any help would be great

[peterdudfield]

Yea, if you fork and run github actions yourself, thats probably the best way to do it

[laurenceandrews]

Sorry, going very slowly on this with Christmas activities going on.

Upon cloning I immediately ran into some failing tests, so I need to see to that before I can analyse the speed issue.

Any idea if the failing tests are a common/known issue before I look deeper into them?

[peterdudfield]

Which tests are failing? You might need to log in to huggingface, but I'm not totaly sure

[laurenceandrews]

Thanks, here's what I've done to consistently get to this impasse:

• Clone from scratch
• Create a venv using Python 3.11.0
• Run conda install -c conda-forge pyresample and pip install quartz-solar-forecast
• Run pip install pytest fastapi huggingface_hub
• Run huggingface_cli login and entered my write access token

I got the following failures, which all seem environment-related. Let me know if there are any specific setup steps you can think of that I might have missed.

test_run_forecast (FileNotFoundError):
Problem: It looks like the test is trying to load model-0.3.0.pkl, but this file is missing from the path open-source-quartz-solar-forecast/venv/Lib/site-packages/quartz_solar_forecast/models/. I only see model-0.4.0.pkl in that folder within venv, but I see model-0.3.0.pkl and model-0.4.0.pkl in root > quartz_solar_forecast > models. Anything I should know about setting up a venv correctly with this project? Quick fix: Manually copying model-0.3.0.pkl into the venv folder fixes this, but I'm not happy with that solution!

test_run_eval and test_get_pv_metadata (EmptyDataError):
Problem: Both tests fail because no columns are found in the metadata.csv file. Again, I'm slightly suspicious that pv.py, which contains get_pv_metadata, sits within the venv, whilst the generated metadata.csv sits in the project's root > data > pv directory.
Potential fix: Venv sync/setup fix or manual download required?

test_get_pv (OSError):
Problem: Similar to above, the test attempts to load a NetCDF file (data/pv/pv.netcdf) that I can see also exists but is empty.
Potential fix: Venv sync/setup fix or manual download required?

test_get_file_path (ValueError):
Problem: The format string used for generating file paths is invalid, due to platform-specificity.
Proper fix: I've updated test_file_path.py to use an f-string for platform-independent formatting.

[aryanbhosale]

Is there a way to speed up CI tests? Locally they are a lot quicker for me? Investigate what is taking a long time in CI

@peterdudfield which one of these?

[peterdudfield]

The pytest.yaml is slow in CI. For me they are also quick locally

[peterdudfield]

Still taking 20 mins, I think we need to do some sort of caching in the CI

[Dakshbir]

Is this issue still open for contributions? @peterdudfield sir

[listopadiya]

FI @peterdudfield @zakwatts I've done a small investigation. Caching would definitely not hurt overall, but what is really slowing down the test, is getting weather data from open_meteo - here is a piece of log, illustrating the issue (I've added some logging in the fork):

If we mock at least some of those web requests, it could help. Would be glad if you assign me to the issue 🙂