Reporting of results that can be validated by others; with code that is reproducible

[tomasvanoyen]

On the one hand, I truly applaud the open climate fix initiative in general. On the other, I have spend a fair amount of (spare) time trying to reproduce any result; and it turns out not to be straightforward - to say the least.

One of the major blocking issues comprises the fact that the NWP used (I believe across all repo's) stems from MetOffice which is a closed data source. As such, reported results cannot be reproduced, and hence the code cannot be validated. This is major drawback on the appeal of the work.

I believe it would be a major step forward if an effort is also taken to provide code/data/experiments that are all readily open and therefore can be directly reproduced and validated.

For instance, all experiments reported in this repo at "exp_reports/013_satellite" comprise NWP source data. It would provide a major step forward if there would also an experiment with only satellite data. The google bucket provides open access to that data source, and hence other developers can pull the code, run that experiment and at least be sure that the code works as reported.

Best regards,

Tomas

[tomasvanoyen]

Here goes my result of

1. making a *.nc file with pv data following psp/clients/uk_pv/README.md
2. running `psp/scripts/train_model.py --exp-config-name sat_only_original_pv -n sat_only_original_pv
3. putting the NWP data source None
4. evaluating the results with experiment_analysis.ipynb

[peterdudfield]

Thanks @tomasvanoyen for all this. Did you have some Icon and Satellite results aswell?

[tomasvanoyen]

Hi @peterdudfield ,

I made some runs also to compare between with and without satellite input:

Specific:

• making a *.nc file with pv data following psp/clients/uk_pv/README.md
• running `psp/scripts/train_model.py --exp-config-name only_original_pv -n only_original_pv
• putting the NWP data source None and Satellite data source to None
• evaluating the results with experiment_analysis.ipynb

The results are a bit strange w.r.t. the role of satellite images. Following the steps, I mention above you should be able to reproduce them. I would be interested to hear if I made a mistake somewhere (hopefully ! ) or something else is going on ..

[peterdudfield]

Thanks @tomasvanoyen this is really interesting. Really appreciate you doing this.

The result I have with satellite are here and I agree, something doesnt match up.

Your error bars look alot bigger than, mine, so just wondering with how many samples you trained your model?

[lorenpelli]

@tomasvanoyen , thanks for this. It is really helpful.

I also have that type of results. I'm only using the existing uk_pv.py configs in which I only put the NWP data source to None in get_data_source_kwargs()

The model is then just using an history of PV output and the last available image from Eumetsat where we do the mean of irradiance in a 0.5 degree lat,long square around the pv_id locations.

So thinking about it, it makes sense that the error increases fastly through the horizons as there is no info regarding the future. The NWP could provide some information for each horizon but we don't include that here. @peterdudfield, the big difference here is that Tomas does not include NWP data.

Note that adding the Open Meteo data from the API is helping significantly and plays a similar role as the NWP.

[peterdudfield]

Thanks @lorenpelli thats very interesting, do you have any plots to show some results?

Open Meteo is indeed userful for some point forecasts.

I seem to remeber @tomasvanoyen you said you ran it with some ICON data? Do you have the results for this?

[lorenpelli]

Sure, here are my results:

uk_pv_sat_only: NWP set to None, keeping only satellite and PV outputs data.
uk_pv_sat_openmeteo: replacing NWP with Open Meteo.
The big flaw of this training is that we don't have the predicted Open Meteo values for each time 𝑡 we only have the "actual" values. As far as I know, there is no archive of historical predicted values. This explains the "flat" MAE across time.

It's interesting that the satellite data brings nothing (even in the short term) when using Open Meteo data.

[peterdudfield]

Very interesting. Yea open meteo unfortunately doesn't give historic, or maybe it does now but you have to pay for it.

Silly question, but is it worth checking the values of the satellite variables that go into the model, and just checking they are not Nans or filled in to zeros?

[lorenpelli]

Here is an example of feature fed into the model (including Open Meteo variables):

example_feature.txt

[peterdudfield]

Here is an example of feature fed into the model (including Open Meteo variables):

example_feature.txt

Looks like it has real satellite numbers and not nans in there

[peterdudfield]

How many examples did you train with?

One thing that you might want to try, is restricting just to 8 hours. The full 48 hours model might struggler to fully understand the full satellite usefulness. This is what I did here

I shoul know this, but is there a feature that says what horizon it is predicting? We might need this to help the satellite data improve the early horizons?

[lorenpelli]

Thanks for the advice, I'll try the training on 8 hours. I was using all the default parameters from the current repo. I keep you in touch.

The forecast_horizons variable is indeed in the feature.

[peterdudfield]

Thanks @lorenpelli

[lorenpelli]

Here is the evaluation of Satellite+OpenMeteo for 8 hours horizons.

The range of MAE is very similar to what is described in https://github.com/openclimatefix/pv-site-prediction/tree/main/exp_reports/013_satellite#backtest-using-satellite.