Writing to the database is a major bottleneck

[dfulu]

Below shows the timeline of a recent run of this app. The x-axis is seconds since the task was triggered, and it shows when different steps in the app start. The biggest piece is writing the results to the database which took a full 5 minutes.

This motivates trying to speed up pushing to database

Extra notes:

• In this run we were using the 15-minutely satellite data, so we needed to interpolate the satellite to 5 minutely. This took more time than when we use the RSS
• In this setup we are running 9 models in total including some which use cloudcasting as inputs
• This was run with 2 vCPU cores and 12GB RAM
• This is from the current cloudcasting branch

[peterdudfield]

Some solutions

• speed up the save method, check it isnt doing anthything funny
• dont save gsp results for some model
• run save in parrellel in the app
• ....

[utsav-pal]

Hi....... I love to resolve it , if u can assign me this

[peterdudfield]

Ive put this on hold, as potential new Data Platform should solve this