Add support for Python 3.12

[Clemo97]

Describe the bug

Open-Source-Quartz-Solar-Forecast is dependent on pv-site-prediction. Some dependencies within pv-site-prediction are out of date and not supported by Python 3.12.

To Reproduce

Steps to reproduce the behavior:

1. Make sure you have Python 3.12 installed and clone Open-Source-Quartz-Solar-Forecast
2. Run pip install quartz-solar-forecast

[peterdudfield]

@Clemo97 did this work?

[peterdudfield]

this is needed for openclimatefix/open-source-quartz-solar-forecast#93

[peterdudfield]

This might need chaning - https://github.com/openclimatefix/pv-site-prediction/blob/main/pyproject.toml#L13

[zakwatts]

pandas-stubs should now be able to be updated as well: pandas-dev/pandas-stubs#625

[Clemo97]

@Clemo97 did this work?

It did but unfortunately my computer died before I could push my changes😭 😭, hence why I never made a PR. All that's required is running the "poetry update" commands check poetry docs for that which will fail on some dependencies which you then have to manually adjust their version to one that is compatible to Python 3.12.
Pandas-stubs is just the first dependency that's incompatible but there are more, I found like 3 others.

[zakwatts]

Ahhh I do hate it when that happens! Okay, I'll have a look into this again now. Thanks for the effort you have put in!

[jamescrowley]

Just hit this myself - any update re getting #124 merged? Happy to support if needed.

[Clemo97]

no update yet, this issue is actually bigger than it looks.
I choose to downgrade my python version to 3.11.
You can look in to it if you like

[gilbertgong]

I've started to look into this. I've fixed two minor bugs, with 3 tests still failing. Here's a summary (from chatGPT) of the current one I'm looking at, which isn't really "fixable" it seems:

---

Summary of Failure on Python 3.12 with scikit-learn 1.3.2
While testing under Python 3.12 with updated dependencies, the test test_old_models[model_v8-expected0] fails due to a binary incompatibility when loading the model_v8.pkl model file.

This model was originally trained and pickled using scikit-learn 1.1.3, but the environment now uses scikit-learn 1.3.2. During deserialization, scikit-learn throws the following error:

ValueError: Buffer dtype mismatch, expected 'intp_t' but got 'unsigned int' in 'const node_struct.feature_idx'

This is caused by changes to the internal memory layout in scikit-learn's compiled components (particularly for HistGradientBoostingRegressor), which are not guaranteed to be stable across versions. As a result, the model cannot be reliably loaded in newer environments.

[simlmx]

@gilbertgong If you want to bump the version of scikit-learn, it seems like this means losing backward compatibility of older models. In other words models trained with the older scikit-learn and saved (as pickle files) won't work with the new code.

@peterdudfield I'm guessing these older models are not used in production? If not and if models would be re-trained with the new code anyway, then it's probably fine to lose the backward compatibility.

Then the fix would simply be to remove this file
https://github.com/openclimatefix/pv-site-prediction/blob/86751306f64e4225a56dcb5ffa01a35d2a7f6c11/psp/tests/fixtures/models/model_v8.pkl

And corresponding lines in the test here:

pv-site-prediction/psp/tests/models/test_load_models.py

Line 14 in 8675130

"model_v8": [77.446404, 78.650178, 79.028029, 96.746425, 95.17781],