Building a (static) binary that runs "everywhere"

[dufferzafar]

Has anyone tried building a pex file of visidata?

https://github.com/pantsbuild/pex

https://pex.readthedocs.io/en/latest/buildingpex.html#using-pants

This will allow me to just copy a single file to run visidata on remote boxes that have varying python versions.

[ajkerrigan]

It's a neat idea. While I tend to stick to pip/pipx installs, I've fiddled with a few different ways of packaging visidata. For pex, this seems to work pretty well:

pex visidata setuptools --console-script visidata -o vd.pex

Or from a local checkout:

pex . setuptools --console-script visidata -o vd.pex

Where we need setuptools because otherwise references to pkg_resources fail, and --console-script visidata tells the resulting pex file to launch this at startup. I'd be curious to hear if you have much luck with this or run into any issues specific to pex!

[dufferzafar]

Yeah, I've tried this and it works well for the latest visidata with Python 3, but a few of the boxes I need vd on are Python 2 (some are 2.6 😢 )

I tried this: pex --python=python2 setuptools visidata -c vd -o vd.pex

but running vd.pex gives the error:

Could not invoke script /apps/infrafs2/szafar/.pex/installed_wheels/e25855bb54e9b895ffd57339ca6edb966d78568496b26e0b62d7b0d295b6449d/visidata-0.98.1-p
y2-none-any.whl/.prefix/bin/vd: caught OSError(2, 'No such file or directory') while trying to execute `[u'/apps/infrafs2/szafar/.pex/installed_wheels
/e25855bb54e9b895ffd57339ca6edb966d78568496b26e0b62d7b0d295b6449d/visidata-0.98.1-py2-none-any.whl/.prefix/bin/vd']` while trying to execute `[u'/apps
/infrafs2/szafar/.pex/installed_wheels/e25855bb54e9b895ffd57339ca6edb966d78568496b26e0b62d7b0d295b6449d/visidata-0.98.1-py2-none-any.whl/.prefix/bin/v
d']`

Tried running the script directly as well, but then I get:

> python2 /apps/infrafs2/szafar/.pex/installed_wheels/e25855bb54e9b895ffd57339ca6edb966d78568496b26e0b62d7b0d295b6449d/visidata-0.98.1-py2-none-any
.whl/.prefix/bin/vd
  File "/apps/infrafs2/szafar/.pex/installed_wheels/e25855bb54e9b895ffd57339ca6edb966d78568496b26e0b62d7b0d295b6449d/visidata-0.98.1-py2-none-any.whl/
.prefix/bin/vd", line 139
    vd().status = print
                      ^
SyntaxError: invalid syntax

[ajkerrigan]

Gotcha, that's no fun :-/. On the plus side, this is a use case PyOxy is trying to make easier. It provides single-file Python 3.9 binaries here. I ran an experiment like this and it seemed potentially useful:

• Build a vd pex file using Python 3.9
• Download a PyOxy binary for Python 3.9, and extract/rename it to python3.9 in the same directory
• Run PATH=$PWD ./vd.pex

That seemed to run the visidata pex file using the PyOxy binary. Still a bit awkward, but maybe friendlier than needing native Python 3 installs on your target machines or going through other packaging tools 🤔 . Worth a shot!

[dufferzafar]

Thanks for PyOxy. I hadn't followed the PyOxidizer project for some time and didn't know about it.

I downloaded python3.10 binary and created a pex file like so:

pex --python-shebang='#!python3.10' setuptools visidata -c visidata -o vd.pex

This would automatically use the python binary from the same path as vd.pex

BUT!

PyOxy binaries are linked against GLIBC, and old boxes that are running on Python 2.6 have VERY OLD glibc. So i can't even run the python3.10 binary there:

❯ python3.10
python3.10: /lib64/libc.so.6: version `GLIBC_2.13' not found (required by python3.10)
python3.10: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by python3.10)
python3.10: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by python3.10)
python3.10: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by python3.10)
python3.10: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by python3.10)

[luser]

I saw this thread and was going to suggest PyOxidizer as a potential solution! Greg is a former colleague of mine, it's a really solid project. The original motivation for it was to make it easier to distribute binaries of Mercurial (which is primarily implemented in Python). The secondary motivation was to allow implementing portions of Mercurial in Rust for performance reasons. (I'm not sure if that's a direction this project would be interested in, but processing data in a performant manner is something that Rust is extremely well-suited for.)

[ajkerrigan]

Oh good call, --python-shebang during the build step is a smarter way to do that 😅 . As for handling the super old glibc... doh. The only other ways I've fiddled with portable visidata in the past were as a docker image or compiled with nuitka. I wouldn't assume that either of those paths would be any smoother, given the issues you've hit so far :(. Hopefully someone else has some good ideas 👀 .

[dufferzafar]

@ajkerrigan @anjakefala @saulpw Do you think it would make sense for Visidata to also be bundled as an AppImage?

I just saw another python app doing this: https://github.com/mxmlnkn/ratarmount/blob/master/.github/workflows/appimage.yml

[ajkerrigan]

It can't hurt to try. I've used nuitka for building python tools before, and didn't realize until very recently that standalone binaries built with the --onefile --standalone options use AppImage by default.

There are a bunch of other options to control linking and embedding, maybe you'll find a magic combination that works in your case. I have no useful ideas though sorry, and "works on my machine" wouldn't exactly be a useful test 😅 .

[dufferzafar]

So I tried compiling visidata with nuitka & it works!

There are still some issues ofcourse:

1. Visidata by default has a lot of loaders - some of which depend on pandas, numpy & a bunch of other modules - which were throwing all sorts of errors. So for a first cut, I disabled a bunch of loaders ( by removing the import line from visidata/__init__.py ) keeping only csv, sqlite, fixed_wdith, http, html, markdown, xml, yaml loaders. These are the files that I mostly deal with, so it's fine for me.

2. The resulting AppImage still doesn't work on the old box because of GLIBC issue, but I think I'll be able to fix this by doing what XXH is doing - compiling statically with musl. Portable binary with musl Nuitka/Nuitka#652

Will post updates on this thread.

[dufferzafar]

Posting some threads here for future reference:

1. There's another appimagetool written in Go that links with musl : Statically linked runtime AppImage/AppImageKit#877, https://github.com/probonopd/go-appimage/releases/tag/continuous, https://github.com/probonopd/go-appimage/blob/master/src/appimagetool/README.md

2. Numpy / Pandas based loaders should work with --plugin=numpy flag of Nuitka. Need to try this. https://github.com/Nuitka/Nuitka/blob/develop/Standard-Plugins-Documentation.rst

3. The suggested approach by Nuitka devs is to build on the oldest box you can: Create one portable statically linked executable file using Nuitka Nuitka/Nuitka#903 (comment)
   So I'll try to do this on an old RHEL 6 box under a conda environment perhaps.

4. Building a static libpython (that would then link via nutika) will require fiddling with the modules setup ( xxh does it too ) - https://github.com/python/cpython/blob/main/Modules/Setup

[ajkerrigan]

Slick, static linking and musl seems smart. Looking forward to hearing how it goes!