IDistributedCache only option

[brianmat]

I like all of the functionality currently provided within FusionCache, but I would like the ability to turn off the memorycache option.

When using Azure Functions I would prefer to just go straight to Redis and not have the overhead of memorycache due to their stateless nature. I still want to have the fallback value capability in the event Redis is unavailable, but I just don't need the extra step of dealing with memorycache.

This would also help with longer-lived data which could change very infrequently but is accessed often. If I have a long TTL on a value I could have an instance where a load-balanced environment could be out of sync. Let's say I have a TTL of 90 minutes on data but due to a production problem I need to update the current value in Redis. I have no way of evicting the current memorycache values even though Redis has a more current (and correct) value.

This does touch a bit on your backplane idea, but in that case, I would still like to have a preferred cache type for a value. I think this could go in the FusionCacheEntryOptions as a PreferredCache property. This value could be an enum of Local or Distributed. This way I could determine which cache to prefer for a value.

So, this could be seen as 1 of 2 proposals to the FusionCacheEntryOptions:

public CacheLocation CacheLocation { get; set; } = CacheLocation.All;

public enum CacheLocation
{
	All,
	LocalOnly,
	RemoteOnly
}
	

public PreferredCache PreferredCache { get; set; } = PreferredCache.Local;

public enum PreferredCache
{
	Local,
	Remote
}

Looking at the GetOrSetEntryInteral I am not sure if one will be easier than the other. If I had to pick only one option I would say option 1 would do what I really want at the moment. Since I know I would be losing performance by going to Redis first then memorycache would have a smaller performance benefit since I can still use the fallback option.

What are your thoughts?

[jodydonetti]

Hi @brianmat thanks for your interest in FusionCache!

Your issue is very detailed and well thought out, and it covers a scenario I'm not currently using a lot (Azure Functions) so there's a lot of food for thought.

I'll take some time to think about it and see if I can come up with a reasonable path forward and a good design that makes sense.

I'll update you asap.

[arnoldsi-vii]

Hi @jodydonetti , Any updates on that?

[jodydonetti]

Hi @arnoldsi-vii , I've been quite busy with finishing the design and impl of the new backplane feature, which I finally released yesterday as a first alpha release 🎉

All of this took way more time and effort than expected to be able to support all the use cases that emerged from the community.

Anyway, I'm saying this because I hope to be able to finally release the next official version very soon (hopefully next week 🤞), and after that I will go over the backlog which includes this very issue, too.

Thanks.

[arnoldsi-vii]

I really like how this library progressing and In a future I probably will be able to add some PRs.
My team currently debating between EasyCaching Library and yours (I do like yours :))

Our platform Using mostly using Serverless where local cache is not really needed due serverless limits, but also EC2 machines where local cache with distributed do a great job

[jodydonetti]

I really like how this library progressing

Thanks, appreciate that!

My team currently debating between EasyCaching Library and yours

If it can be useful youcan find a comparison here.
Also, if they find some wrong informations please tell your colleagues to let me know so I'll update that.

(I do like yours :))

Ahah thanks 😬

Our platform Using mostly using Serverless where local cache is not really needed due serverless limits, but also EC2 machines where local cache with distributed do a great job

Yeah the serverless scenario is something I'd like to support more. I hope to be able to move on very soon, after the final release of the backplane.

[arnoldsi-vii]

@jodydonetti thank you.
I think for server less scenarios it’s better to save only on distributed cache and have fallback to memory cache for short periods.
Basically switch places.

few ideas for future:

• Circuit breaker - with polly it can be very easy implementation
• Only distributed cache option (not sure it worth developing, because people can use default one)

[jodydonetti]

I think for server less scenarios it’s better to save only on distributed cache and have fallback to memory cache for short periods. Basically switch places.

Ok, help me to understand more please.
If it saves only on distributed it means the memory cache would be empty, and if it is empty it means it cannot be used as a fallback. Or am I missing something?
And in case we are talking about both memory + distributed again, now that there is a backplane (which would synchronize all the memory caches) what would be the benefit of always getting data from the distributed first? It seems to me it would be always be slower.

few ideas for future:

• Circuit breaker - with polly it can be very easy implementation

I thought about that initially, but I preferred to avoid an extra dependency.
Do you think the current impl is lacking somehow?
If so, I may think about dropping my own impl and maybe just use Polly.

And thanks for this conversation, it's useful to me for collecting different experiences and opinions from the community, and it will help me to shape the future of FusionCache!

[HichamSeffendji]

Hi @jodydonetti , I have a couple of questions regarding FusionCache Circuit Breaker (I can move this comment to a more dedicated page if this is not the right place):

• Is there a docs that provides more details around the current implementation of Circuit Breaker in FusionCache? (e.g. number of errors allowed before going into a break / cooldown, cannot be set by the user but the [options docs]((family pick up) ) does not seem to mention the immutable value used by default).

• Also, is Polly supported automatically (through dependency injection) or would a manual implementation and wrapper around FusionCache requests be required?

Thanks.

[jodydonetti]

Hi @HichamSeffendji

• Is there a docs that provides more details around the current implementation of Circuit Breaker in FusionCache? (e.g. number of errors allowed before going into a break / cooldown, cannot be set by the user but the [options docs]((family pick up) ) does not seem to mention the immutable value used by default).

No there are no docs currently, only something here and in the inline docs (eg: xmldocs via IntelliSense etc).

• Also, is Polly supported automatically (through dependency injection) or would a manual implementation and wrapper around FusionCache requests be required?

FusionCache does not have any dependency on other libs (except for core stuff like MemoryCache).

You are free of course to use Polly inside the factory itself though.

Hope this helps.

[arnoldsi-vii]

@jodydonetti i didn’t checked the current implementation of circuit breaker, I used Polly in our projects and it worked great (Polly has a lot of abilities as you know)

As for serverless scenario. Lambda (aws scenario) lives for a very short time and has memory limit that you can use, therefore occupying memory in lambda for cache seems not a best option, usually redis in cloud environment close to those lambdas (serverless functions).
I know it will be slower, not sure you will really feel the impact tho. Imagine you server million requests from different tenants on same lambda, the cache size can create timeouts or service failures due lambda limitations.
One solution is to limit memory cache, there is a default “SizeLimit” property of IMemoryCache, but it really works odd.

to summarize, in serverless scenario main cache is distributed and backup is memory

[arnoldsi-vii]

@jodydonetti I want to give another example of serverless and would like to hear you thoughts of how the library will operate.

Let's say we have lambda which will run for 500ms. user made first call to save the data. Memory save was 10ms. Then its started to save to distributed cache (in background) which from some reason started to take more than 500ms. This means the cache will never hit the distributed cache data and other nodes will never know about the change and might show wrong data.

One solution to this issue is to disable background factory in serverless scenario, but from what I saw, if it fail to save to distributed cache it only log the error. Maybe you can add fail event (not sure that FailSafeActivation is that event).

Other solution as I mentioned before is to make distribution cache as main cache, but I think it will lose the purpose of this awesome library.

Let me know that you think.

[jodydonetti]

@arnoldsi-vii thanks for the additional info, I'll think about it and let you know!

[brianmat]

@arnoldsi-vii I don't think making distributed cache the main cache would hamper the library that much. In my opinion, one of the strong points of FusionCache is the ability to handle the fallback values correctly. It's easy to implement a simple Redis cache, but there's a lot of thought put into the cache update logic and I think that's very valuable.

Serverless code does add a wrinkle and I think memorycache has a more limited value due to recycling of apps and management of state.

[jodydonetti]

Hi @brianmat and @arnoldsi-vii , I'm getting back to my backlog of things to consider, so here we go.

In general the main observation behind this proposal was that in some scenarios like serverless, the 1st layer (memory cache) can either 1) be avoided or 2) not be so prominent, since the instances where the code runs are short-lived.

A suggested approach may be to just use a distributed cache (IDistributedCache) directly without using FusionCache at all, but some of FusionCache features like fail-safe and soft/hard timeouts would be useful to have there too.

🗓️ The times they are a-changin'

Some time has passed, and in this time some features have been added and changes have been made: because of this I'd like to get back to some of your previous points to re-evaluate them.

The most important ones are the introduction of the backplane (which solves the synchronization problem) and/or the DistributedCacheDuration recently introduced (here and here).

NOTE: I may comment again on some of your points already commented before, but I want to take this opportunity to re-evaluate everything so please excuse me if I repeat myself.

But first, let's get back to the 2 proposals.

2️⃣ The 2 options

Basically 2 possible solutions have been suggested:

1. NO MEMORY CACHE: add the ability to skip/disable the memory cache, and use only the distributed cache
2. CHANGE ORDER: add the ability to change the order of use, to work first with the distributed cache and only then with the memory cache

Let's reason about them a little bit.

Option 1: no memory cache

One of the reasons against simply using a distributed cache instance (IDistributedCache) though is that it's useful to have fail-safe and the whole fallback behaviour. The problem though is that without a memory cache, there's nothing to fall back to in case there's a problem with the distributed cache, so I'm failing to see how that would work.

If it's true that some of the other features like soft/hard timeouts would still work, others would not, or at least not fully. For example the cache stampede prevention would only work partially: while the factory is running you would have cache stampede prevention (so not a lot of database calls) but the same cannot be said for the initial distributed cache calls, all of which will be made (since without a local memory cache there would be nowhere to locally save the data and let other callers use it 🤷).

Option 2: change order

In this case the idea would be to keep the memory cache (so memory will be allocated, etc) but we would first check the distributed cache, and only then we would check the memory cache in case of problems.

Here the problem would be the same as above for the cache stampede prevention related to the distributed cache, since all the calls to the distributed cache would need to be made.

Other than that the points being made were related to local memory caches out-of-sync, and I don't see much value in this approach since the introduction of the backplane and the DistributedCacheDuration mentioned above.

Now I'd like to comment on specific points made by you.

🙋‍♂️ Points made by @brianmat

When using Azure Functions I would prefer to just go straight to Redis and not have the overhead of memorycache due to their stateless nature. I still want to have the fallback value capability in the event Redis is unavailable, but I just don't need the extra step of dealing with memorycache.

As I said without a memory cache there would be nothing to fall back to, in case Redis would fail. How would you approach this in such scenario?

This would also help with longer-lived data which could change very infrequently but is accessed often. If I have a long TTL on a value I could have an instance where a load-balanced environment could be out of sync. Let's say I have a TTL of 90 minutes on data but due to a production problem I need to update the current value in Redis. I have no way of evicting the current memorycache values even though Redis has a more current (and correct) value.

One thought in general is that, after the beginning of this discussion, I've finally introduced the backplane which would really solve the synchronization problem between an updated distributed cache and a local memory cache.

On top of that the recent introduction of the DistributedCacheDuration mentioned above added ability to specify a different duration for the distributed cache so that you can have, say, a local duration of 1 min and a distributed cache duration of 1 hour.

Do you think one of these new approaches (either a backplane or different durations) would cover your scenario?

This does touch a bit on your backplane idea, but in that case, I would still like to have a preferred cache type for a value. I think this could go in the FusionCacheEntryOptions as a PreferredCache property. This value could be an enum of Local or Distributed. This way I could determine which cache to prefer for a value.

I'm open to this idea, but at the same time it may be a lot of work and introduce new behaviours and edge cases to handle, so I'd like to know if you are still of the same idea and exactly why you would still like to specify a preferred cache type with the other 2 options available.

In my opinion, one of the strong points of FusionCache is the ability to handle the fallback values correctly. It's easy to implement a simple Redis cache, but there's a lot of thought put into the cache update logic and I think that's very valuable.

Agree, but I think you would agree that at this point we are not talking about the option 1 (no memory cache) approach, since there would not be a fallback in case of Redis problems, but only about the option 2 (distributed-first-memory-second), am I right?
I'm trying to reduce the scope of the discussion to better analyze it.

Serverless code does add a wrinkle and I think memorycache has a more limited value due to recycling of apps and management of state.

Since I don't have a lot of real-world experience with a serverless approach, I'd like to better understand how much time (I mean the lifetime of each instance/pod/whatever) are we talking about here: seconds? minutes?

I'm asking because in a scenario where the hypothetical serverless endpoints are called frequently I fail to see the benefits of not having an instance that runs for some time, where a local cache can give you a lot of advantages even if we are talking about a relatively small amount of time (eg: think microcaching) including full cache stampede prevention etc.

And if instead these endpoints are not called that frequently, I think the 1st pass on the memory cache would not be that impactful on the overall run time (I think we are talking about microseconds). It would be like micro optimizing a foreach loop when there's a database query that is taking a way longer execution time.

But again, I'm not that experienced with a serverless approach, so if you can help me understand more I would appreciate that 🙏

🙋‍♂️ Points made by @arnoldsi-vii

I think for server less scenarios it’s better to save only on distributed cache and have fallback to memory cache for short periods.
Basically switch places.

To have the fallback on memory we should 1) not exclude memory and 2) also save on memory, otherwise there wouldn't be anything there to use as a fallback. And when you finally say "Basically switch places" that to me seems like a confirmation that you are not talking about option 1 (no memory cache) but instead about option 2 (distributed-first-memory-second), did I understood it correctly?

As for serverless scenario. Lambda (aws scenario) lives for a very short time and has memory limit that you can use, therefore occupying memory in lambda for cache seems not a best option, usually redis in cloud environment close to those lambdas (serverless functions).

Here instead you seem to suggest avoiding memory to consume less memory.

I know it will be slower, not sure you will really feel the impact tho. Imagine you server million requests from different tenants on same lambda, the cache size can create timeouts or service failures due lambda limitations.

If we are talking about millions of requests I'm not sure a serverless approach is the best use of resources, but again I'm not that experienced with the serverless world so if you can explain a bit more that would be really helpful 🙏

Also, in such a scenario the memory cache has a lot of advantages:

• is always there for sure (no network issues can make it "disappear" randomly, no Fallacies of distributed computing etc...)
• is incredibly fast compared to the distributed one
• has the best data locality in respect of where the data will be used (eg: it sits inside your app's memory)

to summarize, in serverless scenario main cache is distributed and backup is memory

Ok so to me what it's clear that you would like is distributed first, memory second (so, option 2).

I want to give another example of serverless and would like to hear you thoughts of how the library will operate.
Let's say we have lambda which will run for 500ms. user made first call to save the data. Memory save was 10ms. Then its started to save to distributed cache (in background) which from some reason started to take more than 500ms. This means the cache will never hit the distributed cache data and other nodes will never know about the change and might show wrong data.

Not really, if I got the example right: if you don't actively enable background processing of distributed cache operations it will not run in the background. Also if you use the backplane the changes will be automatically propagated as soon as possible, so that the other nodes will be automatically synchronized.

One solution to this issue is to disable background factory in serverless scenario

Watch out about the difference between background processing of the factory (which will happen only if you also enable timeouts, and the timeouts actually occur) and background processing of the distributed cache (which will happen only if you set AllowBackgroundDistributedCacheOperations to true, and by default is false).

In any other case the code will wait for everything to finish.

but from what I saw, if it fail to save to distributed cache it only log the error

This was true, but is not anymore: if you really want you can enable the new ReThrowDistributedCacheExceptions option, which is something I've recently added thanks to a community request.
This option is part of the FusionCacheEntryOptions, so it can be set per-call, in a very granular way.

While we are here though I'd like to point out that manually managing distributed cache errors may be a little too much to handle, so think about it.

🏁 Conclusions

If you withstood this giant wall of thext, thanks 😅, I really appreciate your involvement and your contributions!
Please let me know what you think so we can work something out.

[HashTagDotNet]

Directed Cache Store (L2 Cache Key Only or L1 Cache Key Only)

I too would like the ability to have the ability to read/write explicitly to a specific cache, especially writing a key to only L2 (distributed) cache.

We would like leverage some of the FusionCache's serialization, immutability, and other features while at the same time have the option for a little more control in shaping the caching storage of lifetime of certain keys.

Use-case: Consistency Model

We operate on large web farm with stateless applications. Thus if you update your profile or we authenticate you, we update L2 (distributed) cache only so that the next call hosted by a different server can read those changes.

An L1(memory) cache layer on Host2 would not return a newly set value by Host1 since the new value is on distributed cache.

The eventual consistency is sometimes unacceptable since we need real time consistency across all instances running across all hosts.

Use case: Memory grooming

Under large volumes we want to keep memory cache memory pressure to a minimum while using an L2(distributed) cache to reduce loads on our data stores.

In this case, we would like to be able have L1(memory) cache to have a different lifetime than L2(distributed) lifetime. These lifetimes should be able to be specified at the key level as part of the key options; reverting back to cache level defaults if not explicitly specified.

Use case: Performance

In some cases we want to have keys only in memory and not on distributed cache. We would like to leverage FusionCache's serialization, tags, immutability, and other related capabilities while interacting with L1 (MemoryCache).

Re: Fallback and other capabilities

We recognize that by directing a key to have an explicit store that it might impact other capabilities. In those cases, the engineer willingly makes the trade off.

For instance if a key is on an L2(distributed) only cache, and an unrecoverable transient error occurs the library would return null/not-found state and we would simply re-read the value from the database; fall back algorithm would not be in play.

Suggestion/Request

Per-Key Options

This is something we do in our home-grown caching layer, and it the only thing that is keeping us from considering FusionCache
On each each operation provide the ability to have a level specific simple options for instance:

• L1 and L2 lifetime of the key
• L1 and L2 get/set order

Normally we use set of cache level defaults but there are use-cases where we need to tweak settings so we get a logical result such as L1-Lifetime is 5 minutes and L2-Lifetime is 30 minutes. (reducing per-host memory pressure on high volume keys)

On each of our operations, we also have the ability to define which level we are targeting.

(by example only)
class ExampleOptions
{
   TimeSpan? L1Lifetime;
   TimeSpan? L2Lifetime;
   CacheProvider CacheProvider = Level1ThenLevel2;

   public enum CacheProvider {
       Level1ThenLevel2 = 0,
       Level1Only = 1,
       Level2Only = 2
       Level2ThenLevel1
   }
}
_fusionCache.Set("key","value",new(CacheProvider=L2Only,L2Lifetime="00:01:00"));
_fusionCache.Get("key",CacheProvider.L2Only)

Suggestion - Per-Cache Options

Create a way to disable or not provide either L1 (MemoryCache implementation) or L2 (DistributedCache) implementation to a name FusionCache instance.

If an implementation does this, they would understand that certain features (e.g. fallback) might be limited or crippled since they are not applicable.

Alternate Strategy for Per-Cache Selection

As an alternative to an integrated directed cache key option, we might create three named FusionCache instances with no-op implementations:

• Normal with L2(Distributed/Redis) + MemoryCache
• MemoryOnly with L2(no-op IDistributedCache implementation) + MemoryCache
• DistributedOnly with L2(Distributed/Redis) + MemoryCache(no-op IMemoryCache implementation)

This alternate approach has some disadvantages and does not address the per-cache, per-key lifetime capability we are also looking for.

[jodydonetti]

Hi @HashTagDotNet , thanks for the thorough explanation of your scenarios, I really like getting to know other people's realities to maybe extend FusionCache features.

Having said that, I will not implement the features you are requesting, because... they are already there 🙂

Directed Cache Store (L2 Cache Key Only or L1 Cache Key Only)

I too would like the ability to have the ability to read/write explicitly to a specific cache, especially writing a key to only L2 (distributed) cache.

We would like leverage some of the FusionCache's serialization, immutability, and other features while at the same time have the option for a little more control in shaping the caching storage of lifetime of certain keys.

You can use SkipMemoryCacheRead, SkipMemoryCacheWrite, SkipDistributedCacheRead and SkipDistributedCacheWrite.

Total control of each layer.

Use-case: Consistency Model

We operate on large web farm with stateless applications. Thus if you update your profile or we authenticate you, we update L2 (distributed) cache only so that the next call hosted by a different server can read those changes.

An L1(memory) cache layer on Host2 would not return a newly set value by Host1 since the new value is on distributed cache.

The eventual consistency is sometimes unacceptable since we need real time consistency across all instances running across all hosts.

If you are talking about 2 hosts with both the same value in L1 (memory) and L2 (distributed), then there's an update on host1, and you are worried that the data in L1 on host2 will beout of sync, well there's the Backplane for you.

Just set it up at startup and all your nodes will be in sync automatically.

Use case: Memory grooming

Under large volumes we want to keep memory cache memory pressure to a minimum while using an L2(distributed) cache to reduce loads on our data stores.

In this case, we would like to be able have L1(memory) cache to have a different lifetime than L2(distributed) lifetime. These lifetimes should be able to be specified at the key level as part of the key options; reverting back to cache level defaults if not explicitly specified.

You can user Duration to control both L1 and L2 duration automaticaly: but if you want to specify a different one for L2, you can use DistributedCacheDuration.

Longer distributed duration and shorter memory duration? Yup, done.

Use case: Performance

In some cases we want to have keys only in memory and not on distributed cache. We would like to leverage FusionCache's serialization, tags, immutability, and other related capabilities while interacting with L1 (MemoryCache).

Again, you can use the "skip" options mentioned above.

Re: Fallback and other capabilities

We recognize that by directing a key to have an explicit store that it might impact other capabilities. In those cases, the engineer willingly makes the trade off.

For instance if a key is on an L2(distributed) only cache, and an unrecoverable transient error occurs the library would return null/not-found state and we would simply re-read the value from the database; fall back algorithm would not be in play.

If you are talking about fail-safe, you can control it per-key/per-call.

If you want to skip L1 or L2 you can (see above).

If while skipping L1 and going only to L2 you want to ignore transient distributed cache errors, you can set ReThrowDistributedCacheExceptions to false: in that case the database will be called normally.

Suggestion/Request

Per-Key Options

This is something we do in our home-grown caching layer, and it the only thing that is keeping us from considering FusionCache On each each operation provide the ability to have a level specific simple options for instance:

This is strange... FusionCache has been designed exactly like this since the beginning, I'm not sure what you are missing.
If you take a look at the options available, you'll see that except for some very global options, most of them are entry options.

Basically:

• FusionCacheOptions: options for an entire cache instance
• FusionCacheEntryOptions: options for each single entry

Also, please read here about DefaultEntryOptions: they are the foundation of how you can have a very powerful per-entry options inheritance system where you can define some defaults once, and then per each call you can fine tune/override only some specific options, very granularly.

Normally we use set of cache level defaults but there are use-cases where we need to tweak settings so we get a logical result such as L1-Lifetime is 5 minutes and L2-Lifetime is 30 minutes. (reducing per-host memory pressure on high volume keys)

Again, this is standard stuff for FusionCache, it's the way it has been designed: please take a look at the docs first and see if there's something really missing.

The only thing I really see it's missing (if I got it right) is the ability to swap L1 with L2, meaning access L2 first and L1 second. But that would mean always having, for every single call to the cache, a network + deserialization costs, which is not nice.

Suggestion - Per-Cache Options

Create a way to disable or not provide either L1 (MemoryCache implementation) or L2 (DistributedCache) implementation to a name FusionCache instance.

FusionCache supports Named Caches: you can create multiple named caches each configured differently, with different L1/L2, different options and so on.

Want one cache with only L1 and certain default options + a second cache with L1/L2/backplane with different options + a third with L1+L2 without backplane and so on? No big deal, it works.

Alternate Strategy for Per-Cache Selection

As an alternative to an integrated directed cache key option, we might create three named FusionCache instances with no-op implementations:

• Normal with L2(Distributed/Redis) + MemoryCache
• MemoryOnly with L2(no-op IDistributedCache implementation) + MemoryCache
• DistributedOnly with L2(Distributed/Redis) + MemoryCache(no-op IMemoryCache implementation)

No need, it all just works: just remember that when skipping L1 (memory), cache stampede protection may not fully protect you, but this is the only thing to keep in mind.

Please let me know if I misunderstood something, I'm very curious to understand.

Hope this helps.