backintime-qt script uses non-absolute python3 call

[eddgrant]

Hey folks,

This was baffling me for ages, but I had a flash of inspiration which led me to understand what was going on so I thought I'd share what I have found.

At some point recently the backintime GUI stopped working for me. I initially thought this might have been due to my recent upgrade to Ubuntu 22.04 LTS but none of the solutions I found helped my particular issue.

When I run /usr/bin/backintime-qt I got the following error:

Traceback (most recent call last):
  File "/usr/share/backintime/qt/app.py", line 35, in <module>
    import qttools
  File "/usr/share/backintime/qt/qttools.py", line 21, in <module>
    from PyQt5.QtGui import (QFont, QColor, QKeySequence)
ModuleNotFoundError: No module named 'PyQt5'

My initial thought was that I was somehow missing the PyQt5 package so I checked using pip3 freeze | grep PyQt5 and it confirmed that I had PyQt5 installed:

pip3 freeze | grep PyQt5
PyQt5==5.15.6
PyQt5-sip==12.9.1

I then looked at the contents of /usr/bin/backintime-qt and noticed that it has the #!/bin/sh shebang line. Furthermore when it calls out to python (python3 -Es ${APP_PATH}/app.py "$@") it doesn't specify the path to the python binary.

I think what was happening in my case was that my pyenv configuration was being initialised in the process in which backintime-qt was being run. When it initialised pyenv inserts various "shims" in to the PATH environment variable, with the intention of providing access to a specific Python runtime. The backintime-qt script seems to be expecting to use the system python runtime.

To test this I replaced the line which calls out to python with the following, which ensures that it calls my system python runtime:

/usr/bin/python3 -Es ${APP_PATH}/app.py "$@"

Lo and behold this resolves the issue and the backintime GUI now works for me again.

I'm not sure what a "proper" fix might look like here, as I'm aware that hard coding the path to the python runtime might not be portable across all Linux distros. I did wonder if there was a way to ensure the backintime-qt script was run in a "non-interactive" way, in the hope this might avoid pyenv and other similar tools from polluting backintime's environment, but I didn't have any luck when testing this.

Anyway, thought this was worth sharing and hope this helps someone.

P.S. BackInTime is a fabulous tool, much better than Ubuntu's default backup tool. Thanks for the time and effort spent on it! 🙌

[emtiu]

Thanks a lot for sharing! We're still getting settled in with a new maintenance team, but we'll take a look at what's going on, and how your personal hotfix may improve things for everyone :)

[buhtz]

@eddgrant Awesome. Thank you so much for reporting and analysing this.

I'm not well experienced with virtual environments (e.g. pyenv) because I don't trust them. :D But I have an idea how they work. I will try to reproduce this in a VM.

If I understood you correct you run backintime in a virtuell environment? How can this happen? Or somehow the sh run in the environment?

btw: Maybe a /usr/bin/env python3 -Es ${APP_PATH}/app.py "$@" will help here.

@emtiu Didn't we had other issues with importing PyQt5?

When migrating to the "src" Layout (far away in the future) this won't happen because python (or setuptools and pip) do create the entry point scripts them selfs.

EDIT: To improve diagnosis. Can you please add a line before the last line in your backintime-qt script. Modify this part

fi

python3 -Es ${APP_PATH}/app.py "$@"

into this please

fi

python3 -c "import sys;print(sys.executable)"
python3 -Es ${APP_PATH}/app.py "$@"

This should print the path to the used python interpreter binary on stdout.

[emtiu]

@emtiu Didn't we had other issues with importing PyQt5?

From a quick search, that would be:

• no longer relevant: ModuleNotFoundError: No module named 'PyQt5' when opening BackInTime GUI (Pop!_OS 19,04) #1004
• different problem: [CentOS] ModuleNotFoundError: No module named 'PyQt5' #1042

So, this one looks unique to me at first glance.

[buhtz]

I think what was happening in my case was that my pyenv configuration

How did you installed pyenv? Did you build and installed it direct from github? It seems to me not installable via pip.

Reading further into the docu of pyenv it seems to me that it is a bit exotic and works different then other virtenv systems I know. The virtual environment doesn't exists only in a specific project folder but is active all the time no mater where you are in the file system. The shims are activated globally. But maybe I misunderstand here something. Maybe you can bring some light into it.

[eddgrant]

@buhtzz , here's the extra diagnostic info you asked for:

➜  ~ backintime-qt
/home/egrant/.pyenv/versions/3.9.9/bin/python3
Traceback (most recent call last):
  File "/usr/share/backintime/qt/app.py", line 35, in <module>
    import qttools
  File "/usr/share/backintime/qt/qttools.py", line 21, in <module>
    from PyQt5.QtGui import (QFont, QColor, QKeySequence)
ModuleNotFoundError: No module named 'PyQt5'

Looks like its using my pyenv default Python runtime (which it calls "global"), which is currently set to 3.9.9:

➜  ~ pyenv global
3.9.9

[eddgrant]

I think my pyenv installation probably pre-dates the automatic installer. IIRC I installed pyenv simply by cloning the git repo e.g.
git clone https://github.com/pyenv/pyenv.git ~/.pyenv

How did you installed pyenv? Did you build and installed it direct from github? It seems to me not installable via pip.

[eddgrant]

@buhtzz I think your understanding of how pyenv works sounds correct (at least to me!). As I understand it pyenv is initialised when a shell is started, which in turn executes its initialisation file (.zshrc, .bashrc etc) which contains statements to initialise pyenv. Once pyenv is initialised it figures out which python runtime is desired (shell, local or global) and injects the relevant shims in to the environment.

[buhtz]

Referring to your headline I would say it isn't the responsibility of an application to point to the correct (specific) interpreter. BIT says it needs Python version 3 and then take what the operating system (or shell) does offer.

In your case you kind of "hack" a bit and manage multiple python interpreters of different versions beside the operating systems itself. PyEnv "steal" the responsibility from the OS.

IMHO the reasons for that Issue is that you misused PyEnv. Use PyEnv to get the correct interpreter and its site-packages. Then it should work. Maybe PyEnv is able to point to the original OS own interpreter, too.

Background

PyEnv isn't a virtual environment. It is just kind of a Python interpreter switcher enabling you to install multiple Python interpreters of different versions all with there own site-packages. You can switch between them. The selection is globally in the current shell valid. It is not restricted to a specific folder.

My opinion

Because of that I vote for Close.

Consequences

Despite that your report is valuable. It points me to an improvement for our diagnostics module.

[aryoda]

@eddgrant Excellent problem analysis - thanks a lot for sharing your knowledge here!

pyenv explicitly declares to be able to

... change the global Python version on a per-user basis.

I fully understand that manipulating the search path (like virtenv does here)

• has a major impact on every tools that calls python3 without a hard-coded path to the OS-packaged version
  with sometimes fatal results (like non-installed packages) - this is how Trojans try to get called ;-)
• has its uses cases (eg. when you want to intentionally inject a newer python version for testing purposes)

and that users doing this are fully responsible for the impact.

But: IMHO we should prevent starting the wrong (non-OS-packages) python version for packaged BiT versions to reduce support efforts.

I don't know how, but the OS package maintainers could possibly be able to solve this (it is a packaging issue IMHO).

@buhtzz How about discussing this eg. with the DEBIAN package maintainer of BiT as a starting point, AFAIR you have an account there. What do you think?

[buhtz]

I see no need to discuss this with distro maintainers. It is not easy to argument here in my non-native English. 😄

The use case described here violates the concept of responsibilities. A userland application with dependencies to extern components (packages, libraries or an inpterprete like python2/3) never specifies the path to them explicit.
The operating system is responsible to offer the correct interpreter.
The interpreter is responsible to offer the correct packages.

When you "hack" that concept via a python-interpreter-switcher like pyenv or with virtual environments like virtuelevn that you have to take care of that edge case yourself and can't blame the system or the application for that.

This is Issues is not a bug. It is a misuse of pyenv. I assume that @eddgrant has some problems to understand how pyenv should be used. Switch to the correct python interpreter with pyenv and it should work with BIT. How to do this is up to the pyenv user.

It would violate a lot of concepts if BIT would specify an explicit path to the python interpreter.

• In more modern and currently official recommended python project layouts ("src-layout") using setup.cfg or project.toml to setup the project and its "entry points" (starting scripts installed to bin-folders) the interpreter is not specified that way.
• It violates platform independence not only between Linux and Windows or Mac. There are Linux distros that will throw "python3" away in the future (e.g. Debian) and switch to "python" because "Python" is "Python version 3" by default.
• It could be problematic in testing environments (e.g. TravisCI) because they also using special python interpreters.

@eddgrant You should be able to start BIT with your pyenv when you select a python interpreter where PyQt5 is installed in its environment.

I vote for close because this is not a BIT bug but a misuse of on external tool.

But: IMHO we should prevent starting the wrong (non-OS-packages) python version for packaged BiT versions to reduce support efforts.

There is no usual way for BIT to know what is "wrong". But I will add the used python binary path to the diagnostics in the next days.

I don't know how, but the OS package maintainers could possibly be able to solve this (it is a packaging issue IMHO).

They still do because they are responsible for what "python3" in there system means.

[aryoda]

I think @eddgrant opened this issue more with the intention to share his insights and less to urge a fix.

Nevertheless if I grep -i python3 /usr/bin/* I can find a lot of shebangs with hard coded python3 paths so I think
it could be solved by either using an absolute path injected by the package maintainer or a generic "find distro-packaged python" function in the starter script at our (BiT) side.

...
/usr/bin/asan_symbolize-9:#!/usr/bin/env python3
/usr/bin/backintime:python3 -Es $APP_PATH/backintime.py "$@"
/usr/bin/backintime-askpass:python3 -Es $APP_PATH/askpass.py "$@"
/usr/bin/backintime-qt:python3 -Es ${APP_PATH}/app.py "$@"
/usr/bin/chardet3:#! /usr/bin/python3
/usr/bin/chardetect3:#! /usr/bin/python3
/usr/bin/check-language-support:#! /usr/bin/python3
/usr/bin/chrome-gnome-shell:#!/usr/bin/python3
...

I think a simple if in the shell script could heavily improve the situation, eg. (not tested!):

# ...
python_starter="/usr/bin/python3"
if [ ! -f $python_starter ]; then
    python_starter="python3"
    echo "Unexpected python3 installation path detected (virtual env activated?). BiT may not work as expected"
fi

$python_starter -Es ~/dev/backintime/qt/app.py "$@"

Edit: If this shouldn't use the correct python version the package maintainers of BiT will come back to us anyhow ;-)

Edit 2: BiT common (CLI) is also affected by this issue...

Edit 3: Should we also add the path to the active python file to our diagnostics output (which python3)?