old NWP data, check PVnet intraday works

[peterdudfield]

We should check that if NWP is old, but still enough steps in the future, that PVnet intraday still runs.

Actually is this behaviour we want? With very old NWP data, we might want it to fail on purpose, so no bad results are made

[dfulu]

@peterdudfield what is the intended behaviour here? Currently the app checks if the NWP steps required are available. Do you want to put in a hard failure if the NWP becomes too stale even if it has the required steps?

[peterdudfield]

I think this came out of a conversation with the team. Im not sure if im honest. The question might come down to, does PVNet intraday on stale data work better than PVNet DA? Im not sure. Its also might lead to a lot of cases, where if one NWP data is stale, we still allow it, as the model should still use the other NWP data. Or Satellite data wont be stale, so maybe its ok. Yea, im not sure to be honest

[dfulu]

I don't see the case for this. For example, say the last NWP timestamp was 6am, and we set a max staleness of say 6 hours. At 1pm (and after) the NWP is "too stale", so we either use the stale NWP data to run the intraday model now, or rely on the last day-ahead forecast which was made a noon.

So if the NWP data becomes too stale, then we just start relying on a stale runtime of our own forecast. That doesn't seem like an advantage to me. The day-ahead model doesn't use satellite, so we'd drop accuracy and we won't have predictions out to 36 hours ahead of present

[peterdudfield]

Im not set on this, very much a discussion. I think it came out of a data outage? LIke if Metoffice data is down for 12 hours, is it still ok to run PVnet?

[dfulu]

To come back on this, I think the only way we can move forward on this is to do a backtest. We could:

• Pick some period to do the backtest over
• Do a backtest as usual with our current model
• Change the data config to set the NWP dropout to some high value and run the backtest again with the same model. This gives us a backtest using stale NWP inputs
• Compare

This would tell us how much of an impact stale NWP has on our forecast accuracy.

I think the likely result from the above is that stale NWP will make our forecasts a bit worse. But in that case, I think we would probably accept that drop and make our forecasts using the stale NWP anyway. The alternative is that when an NWP goes down, we stop making forecasts and our PV forecast becomes stale. We know that satellite data is really important for our forecast accuracy. So a fresh version of our PV forecast which uses stale NWP and recent satellite is going to be better than a stale version of our forecast which used the same NWP data (but used it when it was less stale) and uses old satellite data