Option to ignore unknown tasks rather than raise TaskNotFound?

[amacfie-tc]

I'm thinking it might be useful to have the option for workers to ignore tasks that they don't have a handler for, rather than raise TaskNotFound and fail the job.

[medihack]

Hi @amacfie-tc. Do you have a specific scenario where this is needed or helpful?

[amacfie-tc]

It would make deployments easier -- at the moment if we add a new task, we have to make sure the workers are updated before the services that could defer the new task.

[ewjoachim]

The usecase makes sense but any change in the algorithm for selecting a task means the DB needs to have all the info, so it would mean the worker needs to send a list of all the tasks it knows everytime we ask for a new job.

It's possible, that could end end up being quite a lot of info to send, e.g. if there are thousand of tasks (though I don't expect most users to have thousands of tasks).

If we were to do that, I think I'd rather add a version parameter when deferring a job, and a max_version (a.k.a version >= x) when we launch a worker, so that you could easily increment versions when you add new jobs and only new workers would fetch them.

Note

we could imagine doing something more generic than this, a way to tag jobs, and to create somewhat arbitrary conditions a worker could apply apart from queues rules.

Note

We have the same issue when deleting a job: you need to make sure all corresponding jobs have been processed before you stop the last worker, so either you'd need a similar mechanism, or a 2 phase release (which would also be the solution to your case) (so version < x

Note

... And when you change the signature of a task (without backwards compat) you need to make sure that old workers only process old jobs and vice versa (so version = x)