Allow --config to be a Python package or a file relative to that package

[lafrech]

Hi.

I'm trying to distribute alembic migration files with my application. I want the user to basically pip install my_app then alembic upgrade head.

I can put the migration files directory inside the app along with alembic.ini (and modify setup.py and MANIFEST.in accordingly), but then, AFAIU, the user will need to

alembic -c /path/to/virtual/env/lib/python3.x/site-packages/my_app/alembic.ini upgrade head

We could improve this by allowing the user to do

alembic -c my_app:alembic.ini upgrade head

The package:file syntax is inspired by Flask but with different semantics. Well, perhaps : is wrong since it is a valid character in a filename. The point is we need to be able to discriminate a file system path and a package path. We could even use a different config flag.
I think this is how script_location accepts package resources (tutorial).

The option could accept a package and assume alembic.ini in package root. But then again, to discriminate a package name from a relative file name, we might have to use a different config flag.

I've seen people discussing on SO about distributing migration files, but I couldn't find the answer to the alembic.ini path issue. I'm pretty sure there's an already existing and recommended way to do this. In this case, I'd be more than happy to get a pointer, as I've been searching for hours. But I don't mean to turn this into a support request.

In any case, I figured the idea above could be worth sharing.

Have a nice day!

[zzzeek]

What database URL is in the alembic.ini and how is that hardcoded like that? Does your application have its own config file that users are configuring where the database goes ? If your application needs to run alembic upgrades I'd assume your users configure the path to their database in some other config file, so assuming that's the case you can put the alembic config there as well, however, in general what applications usually do is they make their own front-end script or command instead of having users run alembic directly. that's what the command api is for.

[lafrech]

What database URL is in the alembic.ini and how is that hardcoded like that? Does your application have its own config file that users are configuring where the database goes ?

The DB URL is stored by the user in an environment variable read in env.py.

I think I got this from a tutorial from Miguel Grinberg. I may have modifed things so if anything there is fishy, blame me first.

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
if config.config_file_name is not None:
    fileConfig(config.config_file_name)
logger = logging.getLogger("alembic")

# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
DB_URL = os.getenv("SQLALCHEMY_DATABASE_URI")
if DB_URL is None:
    logger.error("SQLALCHEMY_DATABASE_URI environment variable not set.")
    sys.exit()
db.set_db_url(DB_URL)
config.set_main_option("sqlalchemy.url", str(db.url).replace("%", "%%"))
target_metadata = Base.metadata

(The sqlalchemy.url contains an unmodified dummy URL I could probably remove. Honestly I copied the tutorial and was happy to get things working with the modification above in env.py.)

If your application needs to run alembic upgrades I'd assume your users configure the path to their database in some other config file, so assuming that's the case you can put the alembic config there as well

Indeed, rather than asking users to set up an env var with the DB URL, I could provide a template config file and ask them to edit it and set the path to that file in an env var. I'd do this already if I had several configuration parameters. I could try to add logic in that config file to define the path to the alembic.ini file, but I don't think that will help with calling the alembic command directly.

however, in general what applications usually do is they make their own front-end script or command instead of having users run alembic directly. that's what the command api is for.

Oh, so that's the way people do. I assumed it was better to expose alembic (widely used and documented command) rather than add a custom layer above it to expose (even a chosen subset of) its features. Now I'm afraid I chose an unbeaten path that may lead to trouble...

---

One thing I could do as a workaround is just write a command the user would execute in its virtualenv that would export the absolute file path to my_app.alembic.ini into e.g. ALEMBIC_CONFIG_FILE env var. Then tell the user to call

alembic -c $ALEMBIC_CONFIG_FILE upgrade head

It is just one more step.

I think the change I propose in OP would solve this but I won't insist about it, especially if it is useful only to cover my corner case.

Thank you for your quick reply. Much appreciated.

[zzzeek]

I would think users using your library jut want to "myapp upgrade" and not really get into all the other alembic revision commands, so in that sense it may be seen as a more friendly facade

[lafrech]

You're probably right. In practice I don't expect many commands to be needed indeed. I could expose upgrade, downgrade (won't be used but oh well) and perhaps revision if only for devs to generate the migration files easily.

I guess I wanted to make things simple for myself avoiding adding a layer that seemed unnecessary but now I get your point of view. Besides it's only a few commands to add.

Thanks again.

Feel free to close if you think this proposal is not useful. I still think it might make sense if only for me as a dev to use alembic on our deployments without the need for a custom command. But I'm not the one coding and maintaining the feature so I won't complain.

[zzzeek]

oh you made it an issue. will put it in "ideas"

[pwnchaurasia]

@zzzeek @lafrech I am having the same problem,

.
├── MANIFEST.in
├── README.md
├── __init__.py
├── aws_wrapper
│   ├── __init__.py
│   └── aws.py
├── build
│   ├── bdist.macosx-14-arm64
│   └── lib
├── db_wrapper
│   ├── __init__.py
│   ├── __pycache__
│   ├── alembic.ini
│   ├── commands.py
│   ├── db
│   └── migrations
├── requirements.txt
├── setup.py
└── utils_wrapper.egg-info
    ├── PKG-INFO
    ├── SOURCES.txt
    ├── dependency_links.txt
    ├── entry_points.txt
    ├── requires.txt
    └── top_level.txt

above is my directory structure.
utils_wrapper is my package and it has aws_wrapper and db_wrapper for all the aws related things and db realated thing.
This i m going to ship as ppackage
and my team mates would install using pip install.

they would provide their own database credentials in .env file and i want to pass it to alembic and run the migration.
I went through the solution provided by @lafrech in StackOverflow but no luck.

When i do db_current
I get thi

INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

db_downgrade or db_upgrade nothing happens
I get this in terminal
usage: db_upgrade [-NsVv] [-h home] [-P password] db_file ..

My code in exactly like @lafrech
My alembic.init file look like this

# A generic, single database configuration.

[alembic]
# path to migration scripts
script_location = %(here)s/migrations
; version_locations = %(here)s/alembic/versions

# defaults to the current working directory.
prepend_sys_path = .

; sqlalchemy.url = driver://user:pass@localhost/dbname
sqlalchemy.url =

# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = console
qualname =

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine

[logger_alembic]
level = INFO
handlers =
qualname = alembic

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

I set my db url from os.getenv() inside env.py

can you please give me hint what could be wrong here.
I want to run migration from where the package is installed.

[pwnchaurasia]

@zzzeek @lafrech Finally I have solved it thank you very much for this awesome package, and @lafrech for hinting the clear way to do this. you guys are absolute rock star.

[charbonnierg]

@zzzeek I would be very interested to have such feature built-in alembic. My usecase is that I have projects where version_locations must be set dynamically. I can do that easily by creating a custom Config object and then calling commands directly through alembic.command API, but I loose the ability to use the alembic CLI.

I tried to come up with some implementation in my own fork of alembic. If you have a bit of spare time, I would very much appreciate any feedback from you.

Do you think it would make sense to add this feature to alembic and would you be willing it to maintain it ?

[zzzeek]

hi -

as for the general idea of config from a package, we're about to merge pyproject.toml support in #1082, so to the degree that that information is inherent in a python package (?), we could support the syntax proposed at the top and support it for pyproject.toml data.

as for "version locations is dynamic" I dont really understand how getting alembic.ini from a package name rather than a file location helps that. Alembic supports custom CLIs and in the next release (tentatively 1.16.0) this would be even easier if you follow the recipe at https://alembic.sqlalchemy.org/en/latest/cookbook.html#custom-commandline .

[charbonnierg]

as for "version locations is dynamic" I dont really understand how getting alembic.ini from a package name rather than a file location helps that.

It helps because alembic config is not taken from a package name from my proposal, but from a factory function which is called at runtime. However I realize it's not clear that --config refers to a callable and not to a config object.

Anyway I was not aware of custom CLI, it's really interesting, but it does not allow generating a custom config without using a config file does it ?

Thanks for your quick reply

[charbonnierg]

Here is an example usage of what I did in my fork:

This is located within my_app.db package:

from alembic.config import Config

def alembic_config(config: Config) -> None:
    """Configure Alembic with the provided config object."""
    # Set up the script location
    config.set_main_option("script_location", "some/path")
    # Set up the version path separator
    config.set_main_option("version_path_separator", ";")
    # Set up the version locations
    config.set_main_option(
        "version_locations",
        ";".join(
            [
                "some/version/path",
                "some/other_version/path",
            ]
        ),
    )

I run alembic with:

alembic --config my_app.db:alembic_config upgrade head

So I'm free to use whatever runtime generated value for version_locations with this setup (even though that's not the case in the example for simplicity)

[zzzeek]

Anyway I was not aware of custom CLI, it's really interesting, but it does not allow generating a custom config without using a config file does it ?

sure it does, in any version of alembic, subclass alembic.config.CommandLine and override CommandLine.run_cmd(). you get the existing Config and the argparse namespace. Change those to whatever you want, then call super().run_cmd()