How to use Alembic in a plugin-based application with separate migrations per plugin?

[alikhd1]

Summary:

• I'm building a modular application with Alembic, where I want to:
• Have a main app with ~20 tables that always get migrated.
• Have N plugins, each with their own models and Alembic migrations.
• Allow each plugin to have its own migration folder and only apply if the plugin is enabled.
• Allow plugins to have foreign key references to main app tables.
• i use a shared Base between my app models and plugins models

I'm trying to achieve this by creating an Alembic setup in my main app and a separate Alembic setup for each plugin inside their respective folders. The main app migrations are autogenerated and upgraded successfully.

However, when I try to create a revision in a plugin, it raises the following error:
Can't locate revision identified by '0845c2f24355'
This is the revision ID of the main app's database version.

How can I achieve this setup correctly? Is there a better way to implement this?

[zzzeek]

the key question is do you want a single alembic_version table or an alembic_version table per plugin ?

[alikhd1]

I think having a separate alembic_version table per plugin is better, because not all plugins will be active. Each plugin can be optionally enabled during the initial setup, so it's more flexible to manage their migrations independently rather than tracking everything in a single version table.

[zzzeek]

i use a shared Base between my app models and plugins models

Is that part necessary? Or at least that they share a MetaData object? it's fine either way but it means you'd need some extra rules for autogenerate to render correctly for certain models.

I would recommend organizing the Base for each model as described here:

class Base(DeclarativeBase):
    pass


class PluginOneBase(Base):
    __abstract__ = True
    metadata = MetaData()


class PluginTwoBase(Base):
    __abstract__ = True
    metadata = MetaData()

that way you can run autogenerate commands for plugins individually without having to filter on the tables in a single MetaData.

for the different directories the most straightforward way to just use the --name feature described at https://alembic.sqlalchemy.org/en/latest/cookbook.html#run-multiple-alembic-environments-from-one-ini-file .

There's other ways to have just one env.py file and select from among several configured versions directories but this would require some more code.

[alikhd1]

I’m using a shared Basebecause my plugin tables have foreign keys referencing the main app's tables.

However, I think the problem is that Alembic finds the database version from the main app, but then tries to locate that revision inside the plugin’s empty versionsfolder, which causes the error.

[zzzeek]

you want separate version tables per plugin. follow the steps in my answer above