buildx: Cannot connect to the Docker daemon at unix

[klinux]

After upgrade to 2.4.0, buildx can't be used, always returning this error:

+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock --dns 172.18.26.95
+ coredns -conf /etc/coredns/Corefile
Detected registry credentials
+ /usr/local/bin/docker version
Client:
 Version:           [2](https://ci.xxxx.xxxx/repos/668/pipeline/131/10#L2)5.0.2
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Thu Feb  1 00:22:06 2024
 OS/Arch:           linux/amd64
 Context:           default
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
time="2024-0[3](https://ci.xxxx.xxxx/repos/668/pipeline/131/10#L3)-19T18:[4](https://ci.xxxx.xxxx/repos/668/pipeline/131/10#L4)6:[5](https://ci.xxxx.xxxx/repos/668/pipeline/131/10#L5)3Z" level=error msg="execution failed: exit status 1"

Log

time="2024-03-20T17:36:46Z" level=info msg="buildkit debug enabled"
+ coredns -conf /etc/coredns/Corefile
+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock --dns 172.18.26.98
time="[2](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L2)024-0[3](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L3)-20T17:36:46.492203016Z" level=info msg="Starting up"
time="2024-03-20T17:36:46.496044138Z" level=warning msg="Error while setting daemon root propagation, this is not generally critical but may cause some functionality to not work or fallback to less desirable behavior" dir=/var/lib/docker error="could not setup daemon root propagation to shared: remount /var/lib/docker, flags: 0x100000: permission denied"
time="2024-03-20T17:36:46.497540593Z" level=info msg="containerd not running, starting managed containerd"
time="2024-03-20T17:36:46.499153296Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=46
time="2024-03-20T17:36:46.553460254Z" level=info msg="starting containerd" revision=7c3aca7a610df76212171d200ca3811ff6096eb8 version=v1.7.13
time="2024-03-20T17:36:46.594851807Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2024-03-20T17:36:46.594925787Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2024-03-20T17:36:46.595567586Z" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1
time="2024-03-20T17:36:46.595679413Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.595872927Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.595907365Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.595953979Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2024-03-20T17:36:46.596030278Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.596269339Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.596869533Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
.:53
CoreDNS-1.11.1
linux/amd64, go1.21.8, 
time="2024-03-20T17:36:46.602615717Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"ip: can't find device 'aufs'\\nmodprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.602675986Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.603156562Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2024-03-20T17:36:46.603206659Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2024-03-20T17:36:46.603377264Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2024-03-20T17:36:46.603460552Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2024-03-20T17:36:46.603492282Z" level=info msg="metadata content store policy set" policy=shared
time="2024-03-20T17:36:46.612598118Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2024-03-20T17:36:46.612731097Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2024-03-20T17:36:46.612768443Z" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
time="2024-03-20T17:36:46.612807998Z" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1
time="2024-03-20T17:36:46.612875179Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2024-03-20T17:36:46.613200415Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2024-03-20T17:36:46.614011170Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2024-03-20T17:36:46.614376781Z" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
time="2024-03-20T17:36:46.614419893Z" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1
time="2024-03-20T17:36:46.614444301Z" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1
time="2024-03-20T17:36:46.614472486Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614497420Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614519046Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614545395Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614578709Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614600820Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614657817Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614690452Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2024-03-20T17:36:46.614782941Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.614818412Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.614852323Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.614882786Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.614908311Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.614941908Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.614962892Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.614992183Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615021250Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615047212Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615076201Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615097255Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615126174Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615162189Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
time="2024-03-20T17:36:46.615220406Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615249660Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.615270064Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2024-03-20T17:36:46.615810639Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2024-03-20T17:36:46.615881655Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2024-03-20T17:36:46.615917723Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2024-03-20T17:36:46.615945611Z" level=info msg="skipping tracing processor initialization (no tracing plugin)" error="no OpenTelemetry endpoint: skip plugin"
time="2024-03-20T17:36:46.616505511Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2024-03-20T17:36:46.616556325Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
time="2024-03-20T17:36:46.616585093Z" level=info msg="NRI interface is disabled by configuration."
time="2024-03-20T17:36:46.617292095Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2024-03-20T17:36:46.617495969Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2024-03-20T17:36:46.617776377Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2024-03-20T17:36:46.617848897Z" level=info msg="containerd successfully booted in 0.067268s"
time="202[4](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L4)-03-20T17:36:47.[5](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L5)64163354Z" level=error msg="failed to mount overlay: permission denied" storage-driver=overlay2
time="2024-03-20T17:36:47.564300842Z" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs
time="2024-03-20T17:36:47.570297624Z" level=info msg="Loading containers: start."
time="2024-03-20T17:3[6](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L6):4[7](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L7).573496651Z" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)`" error="exit status 4"
time="2024-03-20T17:36:47.596636877Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2024-03-20T17:36:47.59790990[8](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L8)Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2024-03-20T17:36:47.5[9](https://ci.xxxx.xxxx/repos/502/pipeline/143/5#L9)7950558Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)
 (exit status 4)
Detected registry credentials
+ /usr/local/bin/docker version

Environment: k8s

[klinux]

The first problem is the need to run with privileged option.

The second problem, iptables don't be used because error: Invalid argument

Create an image and replacing the /sbin/iptables, the command is a symbolic link to /sbin/xtables-nft-multi, to solve the problem, create a symbolic link /sbin/iptables pointing to xtables-legacy-multi, and this solves the problem to start dockerd.

But I think the better solution is to use kaniko

[qwerty287]

Duplicate of #3537 which contains more details