how to provide directories to python programs?

[dckc]

The example for cloudabi for FreeBSD shows passing a directory:

tmpdir: !file
  path: tmp-unittest

I tried to follow this example but I got:

% cloudabi-run /usr/local/x86_64-unknown-cloudabi/bin/python3 < zv.yaml
Fatal Python error: can't create sys.argdata
IsADirectoryError: [Errno 31] Is a directory: 3

Current thread 0x00007fffffffcd20 (most recent call first):
Abort

This is the yaml spec I tried:

% cat zv.yaml
%TAG ! tag:nuxi.nl,2015:cloudabi/
---
stderr: !fd stderr               # Let Python write backtraces to stderr.
path: !file                      # Search path for Python modules.
    path: /usr/local/x86_64-unknown-cloudabi/lib/python3.6
cwd: !file                       # Local (in-development) python modules.
    path: .
args:                            # Arguments exposed as sys.argdata.
    stuff: junk
    cwd: !file
        path: .
command: >-                      # Inline Python script, like 'python -c'.
    from sys import argdata, stderr;
    import main;
    print(argdata['cwd'], file=stderr)

[dckc]

Thanks for the quick fix.

It's not clear to me how to try it out.

I'm running a FreeBSD vm, but pkg install x86_64-unknown-cloudabi-python-3.6.0_31 says "The most recent version of packages are already installed".

I suppose I should be able to check out this repository and build the one package for the one architecture. Here's hoping I can figure it out...

[dckc]

I'm using the FreeBSD-11.0-STABLE box. This environment doesn't seems to supply python in the expected way:

vagrant@:~/cloudabi-ports % python3 build_packages.py python
... build many other packages successfully...

INFO:src.package:BUILD ninja
INFO:src.distfile:CHECKSUM /home/vagrant/cloudabi-ports/_obj/distfiles/ninja/v1.7.1.tar.gz
WARNING:src.distfile:[Errno 2] No such file or directory: '/home/vagrant/cloudabi-ports/_obj/distfiles/ninja/v1.7.1.tar.gz'
INFO:src.distfile:FETCH https://github.com/ninja-build/ninja/archive/v1.7.1.tar.gz
INFO:src.distfile:CHECKSUM /home/vagrant/cloudabi-ports/_obj/distfiles/ninja/v1.7.1.tar.gz
env: python: No such file or directory
Traceback (most recent call last):
  File "build_packages.py", line 66, in <module>
    build_package(target_packages[(name, arch)])
  File "build_packages.py", line 59, in build_package
    catalog.insert(package, version, catalog.package(package, version))
  File "/home/vagrant/cloudabi-ports/src/catalog.py", line 292, in package
    package.build()
  File "/home/vagrant/cloudabi-ports/src/package.py", line 111, in build
    }, self._lib_depends)
  File "/home/vagrant/cloudabi-ports/src/package.py", line 191, in initialize_buildroot
    dep.build()
  File "/home/vagrant/cloudabi-ports/src/package.py", line 64, in build
    self._resource_directory))
  File "<string>", line 9, in build
  File "/home/vagrant/cloudabi-ports/src/builder.py", line 172, in run
    self._builder.run(self._path, command)
  File "/home/vagrant/cloudabi-ports/src/builder.py", line 350, in run
    ] + command)
  File "/usr/local/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['env', 'CC=/usr/bin/cc', 'CXX=/usr/bin/c++', 'CFLAGS=-O2 -I/usr/obj/cloudabi-ports/include', 'CXXFLAGS=-O2 -I/usr/obj/cloudabi-ports/include', 'LDFLAGS=-L/usr/obj/cloudabi-ports/lib', 'PATH=/usr/obj/cloudabi-ports/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/vagrant/bin', './configure.py', '--bootstrap']' returned non-zero exit status 127

Some package info:

vagrant@:~/cloudabi-ports % pkg query %n-%v|grep python
python27-2.7.12
python3-3_3
python35-3.5.2
x86_64-unknown-cloudabi-python-3.6.0_31

I don't expect it's relevant, but I commented out the stuff I'm not interested in building:

vagrant@:~/cloudabi-ports % git diff
diff --git a/build_packages.py b/build_packages.py
index b576dda..6b93633 100755
--- a/build_packages.py
+++ b/build_packages.py
@@ -42,14 +42,14 @@ for filename in util.walk_files(DIR_REPOSITORY):
 target_packages = repo.get_target_packages()
 
 catalogs = {
-    ArchLinuxCatalog(None, DIR_PACKAGES_ARCHLINUX),
-    CygwinCatalog(None, DIR_PACKAGES_CYGWIN),
-    DebianCatalog(None, DIR_PACKAGES_DEBIAN),
+#    ArchLinuxCatalog(None, DIR_PACKAGES_ARCHLINUX),
+#    CygwinCatalog(None, DIR_PACKAGES_CYGWIN),
+#    DebianCatalog(None, DIR_PACKAGES_DEBIAN),
     FreeBSDCatalog(None, DIR_PACKAGES_FREEBSD),
-    HomebrewCatalog(None, DIR_PACKAGES_HOMEBREW, 'http://example.com/'),
-    NetBSDCatalog(None, DIR_PACKAGES_NETBSD),
-    OpenBSDCatalog(None, DIR_PACKAGES_OPENBSD),
-    RedHatCatalog(None, DIR_PACKAGES_REDHAT),
+#    HomebrewCatalog(None, DIR_PACKAGES_HOMEBREW, 'http://example.com/'),
+#    NetBSDCatalog(None, DIR_PACKAGES_NETBSD),
+#    OpenBSDCatalog(None, DIR_PACKAGES_OPENBSD),
+#    RedHatCatalog(None, DIR_PACKAGES_REDHAT),
 }
 
 
diff --git a/src/config.py b/src/config.py
index 6e146ae..b7ca6ca 100644
--- a/src/config.py
+++ b/src/config.py
@@ -8,9 +8,9 @@ import os
 
 # Architectures for which we can build packages.
 ARCHITECTURES = {
-    'aarch64-unknown-cloudabi',
-    'armv6-unknown-cloudabi-eabihf',
-    'i686-unknown-cloudabi',
+#    'aarch64-unknown-cloudabi',
+#    'armv6-unknown-cloudabi-eabihf',
+#    'i686-unknown-cloudabi',
     'x86_64-unknown-cloudabi',
 }

[EdSchouten]

Hi there! I had to fire up the package builds after I submitted that change. That always takes some time, but should be finished now. Be sure to 'pkg upgrade'. To answer to your problem building ninja: building the cross toolchain still depends on certain tools from the host system, like Python 2. Ed

[dckc]

And python27-2.7.12 isn't enough Python 2 to build ninja and such?

[EdSchouten]

I think it uses $PATH to find it. Do you have /usr/local/bin in there?

[dckc]

yes:

% echo $PATH|grep local/bin
/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/vagrant/bin

[EdSchouten]

That's interesting. The idea is that this should normally work. I'll see if I can reproduce this tomorrow. Thanks for reporting this issue!

[dckc]

I'm past the directory issue.

I/we can open a separate issue on rebuilding FreeBSD packages from source.

vagrant@:~ % sudo pkg upgrade
...
Installed packages to be UPGRADED:
	x86_64-unknown-cloudabi-python: 3.6.0_31 -> 3.6.0_32 [CloudABI]

vagrant@:~/ZeroVault % cloudabi-run /usr/local/x86_64-unknown-cloudabi/bin/python3 < zv.yaml
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'main'

[dckc]

Ah... now I see why python27-2.7.12 isn't enough Python 2. It doesn't actually add anything called python to $PATH. The thing to do seems to be:

% sudo pkg install python
...
New packages to be INSTALLED:
	python: 2.7_2,2 [FreeBSD]
	python2: 2_3 [FreeBSD]

[dckc]

Directory fds get converted to integers... How about Path objects instead? See FdPath in https://github.com/dckc/ZeroVault/blob/cloudabi_wsgi/io_pola.py

Or at least: some flag to tell me it was an fd so I can convert to FdPath myself?

--

cloudabi-ports/packages/python/patch-tool-python

Lines 166 to 168 in 758bcea

	+ } else if (stat_result == 0 && S_ISDIR(sb.st_mode)) {
	+ /* Directories must be expressed as raw descriptor numbers. */
	+ return PyLong_FromLong(value);

[EdSchouten]

As @sgielen is also using Python on CloudABI somewhat intensively, let's add him to this discussion as well.

Sjors, with the program_spawn()/program_exec() bits you added some time ago, I think we're now at the point where we can automatically convert file descriptors to any type we want, as long as we ensure they implement .fileno(), right?

[EdSchouten]

Hi Dan,

I had a look at the Python port with @sgielen the other day: it is indeed the case that any object that has a .fileno() operation can be passed to our program_exec()/program_spawn() bindings.

Dan, would you be interested in writing a Python class that you think would be suitable to wrap directory file descriptors? It could, for example, be based on Python 3.6's new Path abstraction. The only requirements this class should have:

• A constructor that accepts a directory file descriptor,
• A fileno() operation.

Apart from that, you may add any utility function you think is necessary. If you give such a class to me, I'll add it to our version of Python and extend the interpreter's startup code to construct such objects.