Patterns to integrate Alembic with an multi-config/database deployment

[jvanasco]

I have a Python application that is mostly built off the Pylons framework.

To create a deployment/installation, one does the following:

• make a dedicated directory, such as data_development, data_production, or data_testing.
• create/edit a config.ini file in that directory

So the structure is roughly:

/myapp
/myapp/data_development/config.ini
/myapp/data_production/config.ini
/myapp/data_staging/config.ini
/myapp/data_testing/config.ini
/myapp/alembic
/myapp/src
/myapp/pyproject.toml
/myapp/setup.cfg
/myapp/tox.ini

I would like to integrate Alembic in a manner through which Users only have to manage a file or section within each deployment directory. Users should never touch files in the root of the app; everything is compartmentalized within each deployment file.

I can't figure out if this is easily doable or even possible. Aside from playing with env.py to dynamically change vars, the closest thing I found in the docs is using the name flag to require editing a central file.

Can anyone point me in the right direction?

Thanks in advance.

[zzzeek]

if these have .ini files in them, those are alembic.ini files? you'd call alembic -c /path/to/config.ini

otherwise where is your alembic config here?

[jvanasco]

Thanks for the quick response. Unfortunately, --name won't work for this, because that toplevel flie is managed as part of the source, and end users shouldn't interact with it. I need to localize defaults on the top level, overrides within the data directories, and the top level needs to have no idea about potential installations. This is why i've just started at this.

[zzzeek]

are you just looking to have an alembic.ini in the base that acts as a global template for any arbitrary directory in the project? this would look like the alembic.ini has additional path token in it

[alembic]
script_location = %(path_to_users_env)s/alembic

these variables can be injected into a Config instance that you can create yourself, or intercept from the exisitng commandline tool. This works with the toml file too, same dictionary of tokens

from alembic.config import CommandLine

class MyCommandLine(CommandLine):
    def __init__(self, prog: Optional[str] = None) -> None:
        super().__init__(self, prog)
        self.parser.add_argument("--user-environment", type=str, help="required user environment")

    def run_cmd(self, config: Config, options: Namespace) -> None:
         # config_args are the interpolation variables in the ini file.  includes "here" by default.   can add more
         config.config_args["path_to_users_env"] = options.user_environment
         super().run_cmd(config, options)


def main(
    argv: Optional[Sequence[str]] = None,
    prog: Optional[str] = None,
    **kwargs: Any,
) -> None:
    """The console runner function for Alembic."""

    MyCommandLine(prog=prog).main(argv=argv)

running looks like

/path/to/my/custom_alembic --user-environment=/path/to/my/environment

the exact code above has one slight concern which is that it assumes the actual config file hasn't been parsed before we get to run_cmd(). that's always been the case but i dont know that we have tests to assert that config_args is safe to populate at that stage.

we can support an option that does this automatically in a manner similar to the "-x" argument, or we can enhance the "-x" argument to automatically place all values inside the config_args dictionary as well. Then it would just look like:

/path/to/alembic -x user-environment=/path/to/my/environment

when the .ini or .toml file refers to a %(token)s that isn't found, it should raise. so the user-enviroment argument would be intrinsically required.

[zzzeek]

OR, we could just add a new argument --here

/path/to/alembic --here=/path/to/my/environment

if you only need to redirect %(here)s

[jvanasco]

This might work! Thanks so much. I'm going to try some stuff now.

At the very least, I found a bug in Alembic (and potentially fixed it) today because of this.

[jvanasco]

This seems to have been easier than I expected. I hack to adjust the env.py a bit (and generated a PR for one edit), but it seems that I can stuff everything I need (for now) into pyproject.toml and then overwrite the sqlalchemy url in each isoloated config file.