Optimizing Message Passing in Geo-Distributed Numaflow Setup: Using Local Redis Instances on Kubernetes Nodes

[carloshkayser]

Hello everyone,

I’m dealing with a specific use case and need your expertise.

Rather than using a Redis cluster as a buffer for message passing between operators, I’d like to set up an independent Redis instance on each Kubernetes node. How can I achieve this?

I’m deploying Numaflow in a geo-distributed cluster, which introduces significant network latency between nodes. Using a central Redis cluster could increase latency as data would be redirected through potentially distant nodes.

I have a diagram that illustrates my requirements.

Could anyone guide me on how to implement this setup?

Thank you!

[vigith]

We support external redis, have you tried https://numaflow.numaproj.io/core-concepts/inter-step-buffer-service/#external-redis?