Cache stampede problem in Controller

[Rotkaf]

Hi,
In my project I have a controller method that reads some data from cache and returns is out.
If value in cache expired it gets data from external service, caches it again and returns to client.
(Let’s assume that getting data from this service is quite long).
But when traffic is very high and many clients want data from cache when it expires all of them send “their” requests to external service simultaneously.

As far as I googled it is “Cache stampede” problem: https://en.wikipedia.org/wiki/Cache_stampede

What I want of course is to make only the first client request to send a request to an external service and all next make wait until the first request retrieved data from external service and put it cache. Only then the suspended client requests should be “woken up” to read from cache already populated.
Additionally, there should be some timeout to wait for the first request to get data.

Does the Play Framework have something built-in like this?

I’ve done some googling of course but I didn’t find positive results.
I’ve found for example: https://github.com/ben-manes/caffeine/wiki/Faq#bulk-loads
but I don’t know if it can be used in the Play Framework project.

Also did some searching by AI’s and got some solutions but none of them seem simple and I don’t know if it is thread safe, etc.

So, has Play Framework any recommended solution?

[jducoeur]

I can't speak authoritatively for Play, but how to handle this is very different for individual-server vs clustered-with-shared-cache environments -- which one do you have?

Mind, I'm not sure that a simple drop-in library for your problem exists. But the answer fundamentally depends on this question: it's much easier to solve for single-node than for distributed.

[jducoeur]

That's my point -- that's the Play documentation I'm pointing to. Not only does Play use it, it's documented as the default cache.

[Rotkaf]

OK, but question is - does Play Framework use it?

Of course I meant: if PF use in its own code "cache stampede" solution from caffeine :) (if caffeine really gives it)

[jducoeur]

Okay, I'm a little confused. The link you gave to Caffeine is specifically for bulk pre-loading of data. Is that what you're trying to accomplish? It's not what your problem statement sounds like, but perhaps some details are missing.

As far as I can tell, that section talks about cache stampede because that's a particularly complicated issue in the context of bulk loading, but bulk loading isn't the solution to cache stampede -- I believe it's just saying that it's designed the way it is (and you should use it with the async API) in order to avoid cache stampede.

It sounds like you're thinking of cache stampede as a problem that is solved by a specific tool, which is incorrect. It's a design issue that naively-built cache implementations are prone to. Good cache implementations don't have that issue; AFAICT, Caffeine (used properly, with the async APIs) doesn't. There isn't a single "solution": it's about designing the whole cache correctly, and using the right APIs from it.

(It's also a thornier problem for clustered servers sharing a common networked cache, which is why the Wikipedia article calls out memcached specifically. It's easier to avoid for a single-node application using a cache like Caffeine.)

Regardless, I feel like the question is a little misaligned. Play is, by design, cache-neutral: it provides a wrapper API that describes that basic core functionality of a cache; it then provides a couple of typical implementations, of which the preferred is caffeine. It also allows you to provide your own implementation (which could be basically anything that runs in the JVM).

All Play "uses" is that wrapper -- the details are up to you. Since the default is caffeine, and I think caffeine is generally designed correctly, I believe it should be resistant to the problem by default.

But remember, all of this is open source. If you're looking for details, I recommend scanning through that. (Which is all I'm doing.)

[Rotkaf]

The link you gave to Caffeine is specifically for bulk pre-loading of data.

Yes, I gave a link that in description has anything I'd found about cache stampede in caffeine. Maybe it caused a little confusion.

Good cache implementations don't have that issue; AFAICT, Caffeine (used properly, with the async APIs) doesn't.

OK. I suppose caffeine doesn't. I was wondering that PF use it - I hope so.
I'll verify my code carefully again to check it's written correct.
As far as I understood, the completely async code should be free of stampede problem, am I right?

all of this is open source. If you're looking for details, I recommend scanning through that.

Yes, I'll do it but of course it's a quite long process and finally it hard to be 100% sure so first I was wondering that anybody has encountered this problem already or is sure that it works good out of the box with correct code.

Play is, by design, cache-neutral
All Play "uses" is that wrapper -- the details are up to you

I suppose it's generally truth but in some framework code there are some PF's own "insertions" for threads executions like for example there: https://github.com/orgs/playframework/discussions/12353, so I was thinking that with stampede problem there are similar behaviors maybe.

To sum it up for now - I'll check my code if it's fully async.

[jducoeur]

As far as I understood, the completely async code should be free of stampede problem, am I right?

I believe so, although there's nothing like doing some testing to double-check. (That is, writing up an endpoint that proxies the slow fetch, hitting that with two simultaneous requests, and confirming that only one fetch happens.)