Shared state across instances

[deangalvin-cb]

👋 Migrating from Discord here:
https://discord.com/channels/742820443487993987/865294205655973928/1336449652177764402

I was wondering if the concept of shared state has ever come up & if Vector has a stance on it.

Rationale: I would love to do rate-limiting on noisy services, however I can't reliably do that as I don't know which container is going to pickup that service's logs. I was thinking it would be lovely to utilize something like redis to share ratelimiting state across instances, however I wasn't sure if that had ever been discussed (if it has happy to move there instead as well!)

It would be really ideal if we could make VRL calls that had the ability to utilize external services like KV stores to share state across instances. This would allow for much more impactful transforms, sampling, and rate limiting.

[From Discord]

Pavlos Rontidis — Today at 4:58 PM
It's a challenging topic since we don't maintain any servers. We have been focusing on exclusively local components. But I can imagine designing some interfaces and allow the users to maintain their own infra. E.g. for a global quota you would need to maintain a key value store.

[deangalvin-cb]

Thats precisely what I was thinking, some interface or even predefined external sources for state storage.

Id imagine it would be useful to store state "conditionally" ie. storing every log that comes through is wildly expensive, but storing 1/100 would be much cheaper & still give valid results.

[pront]

Hi @deangalvin-cb, this is an exciting new area. It seems we are aligned on the very high level idea of an interface to abstract away infra/storage owned by users. A short RFC (so that we can review and commit to the repo) would work best here. One idea I would explore is an enrichment table that abstracts away a key/value store. But there might be more elegant/flexible ways to go about this.

[deangalvin-cb]

Yes! Thats pretty in line with what im thinking. Im not a rust dev by any stretch, but Im happy to tackle a first draft!

Also sorry for the silence was away for a bit!

[pront]

Im not a rust dev by any stretch, but Im happy to tackle a first draft!

This is great. The interface ideas are language agnostic but it can get tricky to understand Vector core internals. Happy to help if needed. We can keep using this thread.

[johnhtodd]

This is now possible with the memory enrichment tables, though you have to get creative with it. This does pretty much what you're talking about in Redis, but does it in memory much faster (though there is no locking because of the way Vector is built for speed, so you can't assume it is perfect.) You can write data into an enrichment table key/value store, and then read it from subsequent events. You could increment/decrement key counters, etc. It also has a TTL but you can just set that to very high, or reset it every time. #21348 It may not be exactly realtime, but it's probably close enough for your rate limiting example. You could use the existing sample and rate limit methods to constrain what goes into the memory write process, but from what I've seen you may not need to - this will depend on your volume and lots of testing. In your rate limit example, you wouldn't use the ratelimit primitives of Vector - you'd just "abort" 1:x messages when you approach some sort of limit. How would you key this? Probably some sort of modulo for epoch time as part of the key. ("In this minute ending in 03, plus the previous minute ending in 02, there were X events for server Y, therefore 80% of the queries that reach this VRL step will be sent to 'abort'") This would be a lookup of Y03 and Y02 as the keys. (careful with those roll events, folks! I'm sure there is a smarter way to do this with epoch math but that is "left as an exercise for the reader") Every event you add to the Y(minute) value, though, and re-write it into the cache. There are race conditions here, but "does it matter"?