Worker: stop polling for tasks, start pushing

[benoit74]

Currently, workers are polling the API for new tasks to start.

This has the consequence that we have significant traffic to the API linked to this polling. It is a kind of hack. And it seems to be a bit unstable under some conditions (some workers seems to take way more time to start a requested task than others).

One might want to consider (again) pushing tasks to worker. We used sloppy in the past but had to remove it for infrastructure issues reasons.

For this task to be ready to be developed, we need to analyze:

• which push solution do we want to implement, and how will it work in our existing infrastructure
• will there be a real "gain" in terms of traffic, since we need anyway to have worker update their running tasks status on a regular basis
• will there be a real "gain" in terms of stability, since push solution is probably a bit more complex than polling

[rgaudin]

• we used zeromq and it worked great.
• Polling was the fallback. At that time we were not reporting progress nor resources usage though but fallback is anyway necessary.
• We could not using when hosted in Sloppy because it requires an open port to connect to. There are other technologies (websocket for instance) that have lighter requirements but are a lot more complex. To be reassessed obviously.
• At some point we removed the server code from the repo and the client one as well.
• polling delay has always been configurable. Default was 10mn and reduced to 3mn 6 years ago (!).
• Using push will allow tasks to start right-away. That's a usability improvement for zimfarm users.
• Beware that the scheduling module was I believe rewrote with the polling in mind (ie. worker's complete status at hand).

My opinion is that push should be lightweight and dumb. It should just send a event to the worker(s) asking them to poll now. Would be a lot simpler to implement and robust.