Extension fails to start: executable file not found in $PATH

[mottetm]

Bug Report

Description

I'm trying to run a zot registry as an extension, I am able to add it to my installer but the extension fails to start with the following error:

Error running Containerd(ext-zot), going to restart forever: failed to create task: "ext-zot": failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "zot": executable file not found in $PATH

The binary is however present:

$ talosctl ls -lH /usr/local/lib/containers/zot/
NODE          MODE         UID   GID   SIZE(B)   LASTMOD          LABEL   NAME
10.0.42.136   drwxr-xr-x   0     0     160 B     6 minutes ago            .
10.0.42.136   drwxr-xr-x   0     0     40 B      6 minutes ago            dev
10.0.42.136   drwxr-xr-x   0     0     80 B      6 minutes ago            etc
10.0.42.136   drwxr-xr-x   0     0     40 B      6 minutes ago            proc
10.0.42.136   drwxr-xr-x   0     0     40 B      6 minutes ago            run
10.0.42.136   drwxr-xr-x   0     0     40 B      6 minutes ago            sys
10.0.42.136   drwxr-xr-x   0     0     60 B      6 minutes ago            var
10.0.42.136   -rwxr-xr-x   0     0     45 MB     15 minutes ago           zot

$ talosctl read /usr/local/etc/containers/zot.yaml
name: zot
container:
  entrypoint: ./zot
  args:
    - serve
    - /var/lib/zot/config.yaml
  mounts:
    - source: /var/lib/zot
      destination: /var/lib/zot
      type: bind
      options:
        - rshared
        - rbind
        - rw
depends:
  - network:
      - addresses
  - service: cri
restart: always

$ talosctl get mc -o yaml | yq .spec | yq 'select(.name == "zot")'                 
apiVersion: v1alpha1
kind: ExtensionServiceConfig
name: zot
configFiles:
  - content: |
      {
          "storage": {
              "rootDirectory": "/var/lib/zot/data"
          },
          "http": {
              "address": "0.0.0.0",
              "port": "5000",
              "compat": [
                  "docker2s2"
              ]
          },
          "log": {
              "level": "info"
          },
      }
    mountPath: /var/lib/zot/config.yaml

The executable is built for the linux/arm64 target and can successfully be run in the container that builds it (alpine based).

Any idea what could be causing this issue?

Thanks in advance.

Environment

• Talos version: 1.9.1
• Kubernetes version: 1.32.1
• Platform: metal arm64

[smira]

Converted this into a discussions, as it's a question.

This error usually means that a shared library required by the executable is missing in the container.

Extensions solve it one of several ways:

• static builds
• include library into the image
• if it's e.g. musl libc, mount it up from Talos host

Either way is fine.

The error can be easily reproduced by copying out the extensions service rootfs as a separate container image and doing docker run on it.

[mottetm]

Thanks for the quick reply.

Well, the file was indeed dynamically linked to musl, which was missing:

$ file zot
zot: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, BuildID[sha1]=8ee50423995595467264f7548159f9e9e55b888f, stripped

But even after adding /lib/ld-musl-aarch64.so.1 as an extra mount, I still get the same error

$ talosctl read /usr/local/etc/containers/zot.yaml
name: zot
container:
  entrypoint: zot
  args:
    - serve
    - /var/lib/zot/config.yaml
  mounts:
    - source: /var/lib/zot
      destination: /var/lib/zot
      type: bind
      options:
        - rshared
        - rbind
        - rw
    - source: /lib/ld-musl-aarch64.so.1
      destination: /lib/ld-musl-aarch64.so.1
      type: bind
      options:
        - rshared
        - rbind
depends:
  - network:
      - addresses
  - service: cri
restart: always

$ talosctl ls -lH /usr/local/lib/containers/zot/lib
NODE          MODE         UID   GID   SIZE(B)   LASTMOD          LABEL   NAME
10.0.42.136   drwxr-xr-x   0     0     60 B      32 seconds ago           .
10.0.42.136   -rw-r--r--   0     0     0 B       32 seconds ago           ld-musl-aarch64.so.1

$ talosctl services ext-zot
NODE     10.0.42.136
ID       ext-zot
STATE    Waiting
HEALTH   ?
EVENTS   [Waiting]: Error running Containerd(ext-zot), going to restart forever: failed to create task: "ext-zot": failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "zot": executable file not found in $PATH (2s ago)

[mottetm]

Absolute and relative paths fail:

$ talosctl logs ext-zot
10.0.42.136: exec /zot: no such file or directory

$  talosctl logs ext-zot
10.0.42.136: exec ./zot: no such file or directory

No other library is needed.

# ldd  zot
        /lib/ld-musl-aarch64.so.1 (0xffffb61c0000)

[frezbo]

/lib/ld-musl-aarch64.so.1 is a symlink probably

[smira]

Please see above, copy your extension rootfs back to / in a Dockerfile, or build in a Dockerfile first, and test with docker run. Once you're comfortable building container images, you can move to Talos. It will be a faster cycle.

[mottetm]

I did, same result:

$ mkdir ext

$ talosctl cp /usr/local/lib/containers/zot ext

$ ls -lh ext
total 88696
drwxr-xr-x@ 2 mottetm  staff    64B May 16 16:47 dev
drwxr-xr-x@ 4 mottetm  staff   128B May 16 16:47 etc
drwxr-xr-x@ 2 mottetm  staff    64B May 16 16:47 lib
drwxr-xr-x@ 2 mottetm  staff    64B May 16 16:47 proc
drwxr-xr-x@ 2 mottetm  staff    64B May 16 16:47 run
drwxr-xr-x@ 2 mottetm  staff    64B May 16 16:47 sys
drwxr-xr-x@ 3 mottetm  staff    96B May 16 16:47 var
-rwxr-xr-x@ 1 mottetm  staff    43M May 16 16:47 zot

# Dockerfile
FROM scratch

COPY ./ext /

$ docker build .
$ docker run -it sha256:... /zot
exec /zot: no such file or directory

[mottetm]

Ok, it worked with:

name: zot
container:
  entrypoint: /zot
  args:
    - serve
    - /var/lib/zot/config.yaml
  mounts:
    - source: /var/lib/zot
      destination: /var/lib/zot
      type: bind
      options:
        - rshared
        - rbind
        - rw
    - source: /usr/lib/libc.so
      destination: /lib/ld-musl-aarch64.so.1
      type: bind
      options:
        - bind
        - ro
depends:
  - network:
      - addresses
  - service: cri
restart: always

Thanks for the assistance.