Reports of Platform being slow

[SAbdin93]

Hi,

We’ve received a few emails reporting that the platform has been running slowly. I’ve noticed the same issue, logging in takes quite a while, and searching for trusts/patients or entering data is also slow.

Would you be able to look into this for us?

Thanks so much,
Saleema

[mbarton]

Correlating this with data from Azure - during working hours the maximum response time does seem to spike up to around a minute, occasionally spiking higher

CPU and memory usage on the container seem to be fine.

CPU usage on the database is relatively high, with a 100% spike that seems to correlate with high latency spikes on the container

The platform metrics for CPU credits consumed and remaining seem to indicate we're not being throttled.

Database memory usage is high but constant:

We have 25GB remaining of the 32GB allocated so I don't think disk pressure is an issue yet.

We are currently running a Burstable, B1ms, 1 vCores, 2 GiB RAM, 32 GiB storage instance. I suspect that is small for our needs. I will try initially bumping it to Burstable, B2ms 2 vCores, 4 GiB RAM, 32 GiB storage to see if the extra core and memory help.

[mbarton]

Recording for posterity that upgrading the database type took 10 minutes! 5 minutes through I started to worry that it was waiting to drain existing connections from the web app so manually terminated it. No idea if that sped it up. 10 minutes to adjust some burstable VM config!

[mbarton]

Back up and running on the new database instance. Loading the org dashboard was still quite slow (#1092) but other pages seem to load fine. I will give it a few hours of organic traffic before having a look at the metrics again to see if the upgrade made any difference.

My no-proof total hunch theories for slower performance:

• Every update to a patient triggers cascading KPI writes in the database at all levels (org, trust etc)
  • Perhaps that starves out other requests?
• The org dashboard is slow (Slow page load on organisation summary #1092)

The extra database core may help us - in theory it can now handle other requests whilst blocked on a slow request. But obviously two slow requests would still starve us out.

[mbarton]

I've popped in to the platform four times throughout today and it certainly feels snappier. I still saw a long load time on the org dashboard when I started it again after upgrading the database but it seems to load faster now.

There's still some whopping great request latencies in there, though on average they seem a bit lower since the upgrade:

[mbarton]

After adding the duration to the request logs it seems the answer to "which endpoints are slow" is unfortunately all of them, sometimes. I can't immediately see one that is consistently slow.

Given the baseline metrics (CPU, memory) etc are now very stable and fairly low, I think we're at the point of profiling code.

What I can see from the metrics is the most common endpoints appearing to take more than 500ms are the endpoints that update the questionnaire.

Django Silk (#1267) looks like a non-starter for running in production but it works fine in local dev so I think I will seed some patients locally and have a look at the traces.

[mbarton]

#1273 was a nice improvement but hasn't affected the general slowness of the platform. Filtering by requests taking longer than 500ms I get a mix:

Filling out the questionnaire:

and some absolutely stonking latency on the KPI calculations

(this is not that unexpected, we did start moving that out to Celery in anticipation of it being slow but that's still massive!)

I don't see similar results locally in my seeded instance. So I'm thinking the next step is to whitelist Django Silk to just slow endpoints I see in monitoring, test them locally to ensure that silk won't crash on them and run it in prod for a bit to gather traces.