Websocket send message to all clients when served on gunicorn multiple worker

[xiki808]

I have encountered an issue where I figured I cannot send a message to all websocket clients connected, but instead only to the clients that have connected to the same worker, when serving the project on gunicorn with multiple workers.

In my project users can connect to a websocket connection for certain features, while using RESTful api for other features. Each time a client connects, I'm storing the websocket client in an array for later use. When one client triggers a feature, I want to send a websocket message to all the clients, but instead only clients connected to the same worker are receiving the message. Later I figured this is happening because in each worker the array of websocket clients is not the same, and it is based on which worker the client got connected to. So far in order to mitigate this is to limit gunicorn wokers down to 1.

Can someone tell me what I can do to be able to send all websocket clients from all workers?

One solution I tried is to have a separate project A, which runs on one worker, and the main project B ( running multiple workers ) will send to the other project A the websocket clients that connect to project B. Then project A receives the websocket client from B and send to all clients when B tells it to. The problem is that I couldn't figure how to send the websocket client object over from project B to project A

[adriangb]

You'll probably want to set up Postgres or some other sort of persistent thing and use that to communicate between all of your hosts (e.g. LISTEN/NOTIFY)

[xiki808]

Thank you for your reply, I am using SQLite for persistent storage currently. Would you be so kind to let me know how in your idea, you would store the websocket client object in the DB? I have tried to convert it to JSON but it's not serializable.

[adriangb]

When one client triggers a feature, I want to send a websocket message to all the clients

When the host connected to the client that "triggers a feature" gets the "trigger" (whatever that is) it sends a NOTIFY message (or puts something on some persistent queue) that all other hosts are listening for. When a host gets that message it propagates it to all of its connected clients.

[alex-oleshkevich]

@xiki808 go with Redis pubsub or streams. You can also check broadcaster project that supports multiple pubsub servers and have an example you look for in docs

https://github.com/encode/broadcaster

[alex-oleshkevich]

I won't recommend postgres for this purpose as it acquires persistent connection to PG which is too costy for web app.

[xiki808]

@alex-oleshkevich this looks promising. Thank you.

If I'm getting it right, broadcaster 'stores' the websocket client in a channel, being a redis data store. Each separate broadcaster instance for in each worker, can then access the whole pool of clients, received from the different workers.

[alex-oleshkevich]

Mostly like this. Each websocket connection subscribes to a named channel on Redis side. Then you can deliver the message to all subscribers by publishing to that Redis channel. All websocket connections subscribed to the same channel will receive the message from the channel.

[febin-micheal]

https://medium.com/@philipokiokio/broadcasting-websockets-messages-across-instances-and-workers-with-fastapi-9a66d42cb30a