Hot reload task function

[ardevik]

Hi All,

In my search i just found procrastinate worker need to be restarted if we add new task function. by any chance i don't search deeply, Is procrastinate have ability to hot reload task function without reload/restart the worker, in my case i will add task queue logic on FastAPI so i just need restart the FastAPI as long as worker know the parent module ?

[ewjoachim]

Hey, thanks for your question!

As far as I know, tasks are looked up by the worker on the app, so whenever a new task is dynamically added to the app in the current process, it will be available to the worker.

I therefore understand your question as "the python code has been modified after it was imported, and I want the new tasks to be seen" ? In that case, I think I mentioned a few details here: #1246

I'm not an expert in Python reloading, so feel free to point any mistake, shortcoming or additional ideas that could be useful.

[ardevik]

yes, i plan to add new python module after the worker already running

i find procrastinate worker need to be restarted if we add new task function/module on my research via GPT, and also find other library

https://taskiq-python.github.io/guide/cli.html#auto-importing
https://taskiq-python.github.io/guide/cli.html#hot-reload

but no official support for postgresql broker, so i come up with this question in procrastinate discussion.

so your answer "whenever a new task is dynamically added to the app in the current process, it will be available to the worker" its mean i can add new python module dynamically right without restart the worker.

[ewjoachim]

so your answer "whenever a new task is dynamically added to the app in the current process, it will be available to the worker" its mean i can add new python module dynamically right without restart the worker.

It all boils down to what exactly "dynamically" means. Also, I'm relying a lot here on Python's default behavior, so the limits we have is the limits Python has and the technical choices we made.

What I mean by dynamically is that if after the worker has started, some code running (e.g. called by a job) defines a new task, it will be available.

The problem you'll have is that by default, Python doesn't reload modules when they're changed. If a running Python interpreter has already imported a file, it won't import it again. Also, even if it did (e.g. using importlib.reload, the tasks referenced in app.tasks would point to the old tasks, not the new ones. And you'd likely get an error when reloading because Procrastinate tries to avoid when multiple tasks of the same name are registered, because more often than not, it's a mistake to override a task. What importlib.reload does is mostly re-execute the code in the module, it doesn't track all uses of objects that were in the old module to try and replace them with new ones. That would be a horrible mess. In practical terms: as far as I know, it's impossible to reload Python modules without restarting the whole interpreter.

That said: you don't need procrastinate to support hot-reloading if you want hot-reloading: procrastinate is a single process, and you can install watchdog and use something along the lines of:

$ watchmedo auto-restart -- procrastinate worker

How is it different from taskiq: taskiq uses a model where they create subprocesses, so they're responsible for their subprocesses. This brings a lot of complexity, but also lets them manage a few things more easily than Procrastinate. Taskiq internally uses watchdog to do that: the main process listens for file changes and restarts the children processes.

Procrastinate doesn't do that (yet). Instead, we consider that the worker is a single process and let the parent application (systemd, docker, ...) handle multiple workers. This makes the mental model much simpler, at the cost of not being able to control the worker process itself. But it means Procrastinate will likely play along well with other software that control the process, such as Watchdog.

[ardevik]

thanks for the pointer to use watchdog