How does psycopg2 finds libpq.5.dylib?

[ccrvlh]

After upgrading to Postgres 15 using Homebrew I can't get psycopg2 to work anymore (I'm avoiding switching to psycopg3 since it gives me some errors with datatypemismatch with str/uuid, so I imagine it might still be early to switch).

What I'm seeing after having upgraded is:

  File ".../.venv/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi
    import psycopg2
  File ".../.venv/lib/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: dlopen(.../.venv/lib/python3.11/site-packages/psycopg2/_psycopg.cpython-311-darwin.so, 0x0002): Library not loaded: /usr/local/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib
  Referenced from: <D837BB9B-1388-3D66-AB42-088D115A0BAF> .../.venv/lib/python3.11/site-packages/psycopg2/_psycopg.cpython-311-darwin.so
  Reason: tried: '/usr/local/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib' (no such file), '/usr/local/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib' (no such file)

This file doesn't exist anymore, even though /usr/local/opt/postgresql@15/lib/libpq.5.dylib does exist, I can't seem to make it actually find it. I've been trying to clean up references of Postgres@14 from my machine, I though I cleaned everything, but obviously I didn't, but I don't know what is missing.

Tried removing PSQL using Homebrew and reinstalling it multiple times, fixed the CPPFLAGS and LDFLAGS, no traces of psql@14 on my path, installed libpq, unlinked it and --force linked with psql@15, restart pc, anyways... Have tried everything I could think of to try to make this work. Any suggestions on troubleshooting this? I've seen quite a few discussions, but apart from the link, reinstall psycopg2, and a couple of other things, don't seem to solve the issue I'm experiencing. Any suggestions are welcome!

[dvarrazzo]

The library is found at compile time. It is unfortunate that brew changes its position, but it's their choice. You could report them the problem. Maybe a symlink in a standard path that doesn't change could be a good idea.

AFAICS you mush reinstall psycopg2 to find the libpq in the new location.

[ccrvlh]

Thanks. I've read the discussion on the brew repo as well. Basic reinstall didn't work in my case, even after rebuilding the venv. It seems --no-binary is necessary, although poetry seems to have some restrictions with it for some reason. I was mostly curious on how potgresql@14 ended up in this error, if it's a fallback location, would it make sense to add other paths as fallback as well? My libpq is currently at /usr/local/opt/postgresql@15/lib/libpq.5.dylib.

[dvarrazzo]

I don't know poetry but I've had many reports that it gets massively in the way. If its behaviour is different from pip install, we will not guarantee that psycopg works correctly. If it uses caches that get in the way, you will have to deal with them.

Psycopg uses the pg_config tool to find the location of libraries and include files at build time. Try running pg_config: do you get a PG 14 or PG 15 location? Hopefully it should have been updated consistently with the location of the libpq; if it didn't, problems may arise there.

[ccrvlh]

The weird thing is that pg_config doesn't have any mention of postgresql@14 on my env. And even stranger is the fact that is has a VERSION = PostgreSQL 16.2 which is not PSQL version I'm running. I think this is an unfortunate combination of the way brew handles this and the logic psycopg uses. Anyways, managed to get it working with the no-binary option in pip since poetry doesn't seem to handle that all that well.