Is it possible to run k3d (using the amd64 k3s image) on an aarch64 colima instance using VZ/Rosetta on an M3 macbook?

[chrisRedwine]

My setup is what's described in the title:

MacBook

System Version: macOS 14.7 (23H124)
Kernel Version: Darwin 23.6.0
Chip: Apple M3 Pro
Total Number of Cores: 12 (6 performance and 6 efficiency)
Memory: 36 GB

Colima

limactl version: 1.0.1
colima version: 0.8.0
command:

colima start \
  --profile=relay \
  --activate \
  --arch=aarch64 \
  --cpu=10 \
  --cpu-type=host \
  --disk=100 \
  --memory=28 \
  --runtime=docker \
  --vm-type=vz \
  --vz-rosetta \
  --mount-type=virtiofs \
  --mount-inotify \
  --network-address \
  --ssh-agent \
  --ssh-config \
  --very-verbose

Docker

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0
    Path:     /Users/chrisredwine/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3
    Path:     /Users/chrisredwine/.docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/chrisredwine/.docker/cli-plugins/docker-scan

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-47-generic
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 27.35GiB
 Name: colima-relay
 ID: 7bd9f1dd-a37b-4ae6-a826-c0eb9acc8478
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: <redacted>
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

K3d

version: v5.7.4
command:

k3d cluster create relay \
  --image rancher/k3s:v1.30.6-k3s1-amd64 \
  --agents 1 \
  --servers 1 \
  --trace

However, k3d cluster creation hangs at:

NodeWaitForLogMessage: Node 'k3d-relay-agent-0' waiting for log message 'Successfully registered node'

The results of k3d node list show the loadbalancer container isn't running:

NAME                 ROLE           CLUSTER   STATUS
k3d-relay-agent-0    agent          relay     running
k3d-relay-server-0   server         relay     running
k3d-relay-serverlb   loadbalancer   relay     created

Inspecting the container logs for the k3d-relay-agent-0 shows these errors:

level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:55094->127.0.0.1:6444: read: connection reset by peer"

This works perfectly when using QEMU instead of VZ/Rosetta, but I'd obviously rather have fast mode.

Anyone tried this setup before, or seen similar errors? Nothing I've searched in terms of issues seems similar.

[abiosoft]

You may get it to work but it is not supported and you'd face multiple quirks.