GraphCast model integration

[Ker-ple]

GraphCast is a weather model released by Google DeepMind which is far more accurate than traditional weather models and has decent accuracy as far as 10 days out. A pre-trained model is open-sourced on GitHub and can forecast on a small machine, so it seems possible to integrate with PirateWeather. What do y'all think?

[alexander0042]

I saw this earlier, but it's interesting to see the progress on it, and would love to get it rolled in! My big question is how fast it is at running the operational model, since that would probably be the best way to implement it? It looks like the operational one can pull in data and produce a forecast, which is perfect, but if it's heavy to run that would be an issue.

The other option is to create a forecast for each requested point "live", as it's needed. This would work if it's a really fast model, but would require some sort of automated way to pull in the input data. Either way, it's a super cool project, and definitely on my radar.

[Ker-ple]

The operational model is quite slow (minutes) for predicting the first 6-hour step, but quickly speeds up for successive steps.

Because the model makes a prediction for 1440 * 721 = 1038240 points on Earth on each step, I'd recommend:

1. Spin up a Fargate machine to make predictions whenever new data is available.
2. Partition the result into files by geographic location and send the files to S3.
3. Send the files from S3 to CloudFront.
4. Whenever a user makes a request, return the geographically relevant files from the CloudFront cache.

This should minimize time spent processing data and return results to users as quickly as possible.

[cloneofghosts]

Just wanted to point out that GFS Graphcast data is available on AWS so you could always integrate that first before hosting the model yourself

[cloneofghosts]

@alexander0042 Looks like someone integrated GFS Graphcast into Herbie https://herbie.readthedocs.io/en/latest/user_guide/tutorial/model_notebooks/gfs.html#GFS-GraphCast but it's not in the latest release atm

[cloneofghosts]

I recently found this AWS dataset which contains a few more AI weather models https://registry.opendata.aws/aiwp/

There's also this https://registry.opendata.aws/noaa-nws-fourcastnetgfs/ and ECMWF released a new version of their AIFS model recently as well.