Make sk-tracer able to horizontally scale #80
drmorr0
started this conversation in
Feature Requests and Future Work
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Right now there's just a single sk-tracer instance that sets a watch on the apiserver. For very large/production clusters this could potentially not scale well. We should do some benchmarking here first (#73), but at some point in the future we may need to make sk-tracer able to horizontally scale.
I think in theory this is straightforward: just have a queue with the apiserver watch, and then multiple instances of sk-tracer that pull things off the queue and record them in their local trace instance (probably keyed/sharded by hash, so that one instance doesn't get the "object creation" event and another instance the "object deletion" event). They could either then all merge to some external/persistent datastore asynchronously, or possibly just do a merge "on demand" when a user initiates an export API call.
This is all a bit hypothetical, since I don't think we're there yet, but wanted to record it for posterity.
Related issue: #79
Beta Was this translation helpful? Give feedback.
All reactions