More ZMQ Streamers #72
Description
Just some things I've been thinking about prototyping soon but I wanted to talk through them first / get your thoughts @bmcfee. This idea isn't fully fleshed out yet.
I've been doing some work with ZMQ at work lately, and I am learning that there are actually various different paradigms ZMQ is designed for. Currently, we are using only the "paired" mode (request/recv).
In particular, I am thinking about creating multiple Streamers, wrapped as a sort of "Worker" in separate python processes, streaming to one central ZMQ receiver, which supplies batches to the training process. It should be possible to have the ZMQ Workers live on other machines, as well, and therefore enabling a sort of "CloudStreamer" through AWS/GCP/etc.
I have this pyzmq example in mind. (Although it's currently unclear to me if we'd want a Queue device or a Streamer ZMQ device).
A first step might be to create something like a ZMQWorkerStreamer, which is a Streamer, but sends to the intermediate Queue-like space. Then, you have a ZMQConsumerStreamer which might operate in much the way that ZMQStream does now, except with external sources.
(This also might be along the lines of the asyncio version we're talked a little about, with the added bonus of external sources).