Rootless podman doesn't find catatonit

[Gianluca755]

What did you do

• How was the cluster created?
  • systemctl --user enable --now podman.socket
  • XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
  • export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
  • export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
  • k3d cluster create

What did you expect to happen

Creating the k3d cluster.

Screenshots or terminal output

INFO[0000] Prep: Network                                
INFO[0000] Re-using existing network 'k3d-k3s-default' (284f4e3c08d1d980bbbed7777cad2d9363f540a9501e0cd0e3ad5467f8d7b840) 
INFO[0000] Created image volume k3d-k3s-default-images  
INFO[0000] Starting new tools node...                   
ERRO[0000] Failed to run tools container for cluster 'k3s-default' 
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
ERRO[0001] Failed Cluster Creation: failed setup of server/agent node k3d-k3s-default-server-0: failed to create node: runtime failed to create node 'k3d-k3s-default-server-0': failed to create container for node 'k3d-k3s-default-server-0': docker failed to create container 'k3d-k3s-default-server-0': Error response from daemon: container create: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory 
ERRO[0001] Failed to create cluster >>> Rolling Back    
INFO[0001] Deleting cluster 'k3s-default'               
ERRO[0001] failed to get cluster: No nodes found for given cluster 
FATA[0001] Cluster creation FAILED, also FAILED to rollback changes!

• What did you do afterwards?
  I'm on Ubuntu 20.04 with catatonit installed at /usr/bin/catatonit and tini at /usr/bin/tini. I configured podman in ~/.config/containers/containers.conf to use

[containers]
init_path="/usr/bin/catatonit"

or

[containers]
init_path= "/usr/bin/tini"

but no luck.

Which OS & Architecture

arch: amd64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /run/user/1202402077/podman/podman.sock
filesystem: UNKNOWN
name: docker
os: ubuntu
ostype: linux
version: 4.4.2

Which version of k3d

k3d version v5.5.2
k3s version v1.27.4-k3s1 (default)

Which version of podman

Client:       Podman Engine
Version:      4.4.2
API Version:  4.4.2
Go Version:   go1.20.1
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_100:2.1.7-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d'
  cpuUtilization:
    idlePercent: 71.45
    systemPercent: 5.34
    userPercent: 23.21
  cpus: 8
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1202400513
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1202402077
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-1059-oem
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 9165910016
  memTotal: 33376772096
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_100:1.8.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.6
      commit: 18cf2efbb8feb2b2f20e316520e0fd0b6c41ef4d
      rundir: /run/user/1202402077/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1202402077/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_100:1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.1
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 88h 10m 55.00s (Approximately 3.67 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 1004315222016
  graphRootUsed: 226401210368
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 102
  runRoot: /tmp/containers-user-1202402077/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.20.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.2

[iwilltry42]

Hi @Gianluca755 , thanks for opening this issue.
I converted it to a discussion since there's no real bug here, as k3d is not tailored to (rootless) podman and just happens to support it due to Podman's compatibility layer for the Docker API.

Anyway, k3d just sets the equivalent to the docker run --init flag. Does that one work for you?

[Gianluca755]

Hello @iwilltry42, unfortunately no:

docker run --init alpine
{"msg":"exec container process (missing dynamic library?) `/run/podman-init`: No such file or directory","level":"error","time":"2023-08-17T06:46:37.102974Z"}

[iwilltry42]

I fear that this should be working for k3d to have a chance of getting this right 🤔
Does podman run --init work though? I have no experience at all with podman.

[Gianluca755]

My bad, I don't have docker, the previous command used podman under the alias docker. The error for podman run --init is the same as the above.

[iwilltry42]

OK, so it looks like something is missing in your setup? 🤔

[Gianluca755]

That's for sure, it's a company device with limitations. I'll go bother the podman community, thanks anyway