Looking for ideas to migrate from 2.15 to 3.X using django integration without service interruption

[ducdetronquito]

Hello everyone !

I use procrastinate 2.15 in production with the available django integration and I trying to plan how to upgrade to procrastinate 3.X without service interruption.

The context is that I do blue-green deployments automatically via Github Actions, with the following steps:

• Service is running in production with code version N (and procrastinate 2.15)
• Apply DB migration of code version N+1 (and procrastinate 3.0.2 migrations) via django's migrate command, applying all pending migrations required for service with code version N+1 to be able to run
• Deploy service with code version N+1 (upgraded to procrastinate 3.0.2), running concurrently with the service with code version N
• Shut down service with code version N
• Enjoy my day

I've read both Migrate the Procrastinate schema and Use Procrastinate migrations with Django docs and I don't understand how possible it is to upgrade procrastinate from say 2.15 to 3.0.2 without service interruption (with my current deployment strategy) because the django migrations contains both the "pre" and "post" migrations.

I'm curious about what was your strategy to upgrade procrastinate from 2.15 to 3.X in the context of blue-green deployments of a django project.

Genuine & newbie question: why is it necessary to pack both "pre" and "post" migrations in the same release if "post" migrations can be applied anytime after the code has been upgraded ? Would'nt it be possible to move the "post" as being the "pre" migration of the next release ?

Example: Instead of having migrations 3.0-pre, 3.0-post, 3.1-pre, 3.1-post, we would have 3.0-pre, 3.1-pre, 3.2-pre, 3.3-pre.

I hope my questions make sens :)

Have a nice day and thanks again for procrastinate which is a very nice library to work with 🌴

[ewjoachim]

It's always a bit complex to handle migration for someone else's DB, so I have to admit I don't have all the answers.

I'm guessing the only realistic answer is that the more specific your needs are, the less likely you'll be able to blindly run third party provided migrations :)

Now to answer you, I guess that for now, what we had in mind is that when you want to have precise control over migration, you don't run ./manage.py migrate but ./manage.py migrate procrastinate 00xx and then you're in control of exactly when the migration runs. Of course, I understand this might play poorly with a fully automated workflow, but the precise answer depends so much on how you run your migrations precisely that it's hard to tell.

That said, there is something we could do: when we release we could release a pre-release (or a post-release), e.g.:

• release 3.2.0 that contains pre-migrations only
• release 3.2.0.post1 that also contains post-migrations

This way, you can specify to your package manager that you want exactly the release without the post release, make your release, and then switch to the post-release on the next version. It's kind of the same as what you mentionned, but instead of the post migrations being delayed to the next release, they're delayed to a sementically equivalent immediately following release.

Genuine & newbie question: why is it necessary to pack both "pre" and "post" migrations in the same release if "post" migrations can be applied anytime after the code has been upgraded ? Would'nt it be possible to move the "post" as being the "pre" migration of the next release ?

It's not necessary per se, as we were doing that before, but we find it confusing both for us to have "pending" migrations that we needed to remember to ship on the next release, and if we don't do another release right away, it meant the schema might stay "dirty" for a long time (unless we resort to doing 2 releases each time which... coincidently ends up being the thing I suggested above, and the same as you suggested.

What can you do for 3.0: I guess one possibility is to temporarily fork procrastinate, create a branch based on 3.0 were you remove the post migrations and deploy this to your servers. Same for 3.1.

The conversation is not closed or anything, I'm welcoming feedback, ideas and how we can move forward.
(also, if anyone wants to contribute the change to the release workflow that splits minor upgrades into normal and post, go ahead :) )

[ducdetronquito]

Thanks for you quick answer and hints!

Having .postX release containing post migrations would surely be welcome and I understand that it's easier to maintain than my proposition to delay "post" migrations to the next release.