'ErrImageNeverPull' with nerdctl as regular user pulling container image on k8s worker node but not as root

[chef-wally]

I have set up a container registry on k8s-container-registry-1 using nerdctl and pushed a custom image to it. However, when attempting to pull the image on k8s-worker-node-1 as a regular user, it results in an "ErrImageNeverPull" error. I noticed that running the pull command as root works, but I would like to make it work for the regular user as well.

Any guidance on resolving the pull error for the regular user would be welcome.

Below are the exact steps I completed in order.

Completed on k8s-control-plane-1 server as a normal user:

$ tee my-node-app/my-node-app-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-node-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-node-app
  template:
    metadata:
      labels:
        app: my-node-app
    spec:
      containers:
      - name: my-node-app
        image: k8s-container-registry-1:5000/my-node-app:latest
        imagePullPolicy: Never
        ports:
        - containerPort: 3000
EOF
$ kubectl apply -f my-node-app/my-node-app-deployment.yaml

Completed on k8s-container-registry-1 server as a normal user:

$ curl -LO "https://github.com/containerd/nerdctl/releases/download/v1.7.2/nerdctl-full-1.7.2-linux-amd64.tar.gz"
$ curl -LO "https://github.com/containerd/nerdctl/releases/download/v1.7.2/SHA256SUMS"
$ sha256sum -c <(grep 'nerdctl-full-1.7.2-linux-amd64.tar.gz' 'SHA256SUMS')
$ sudo tar -C /usr/local -xzf nerdctl-full-1.7.2-linux-amd64.tar.gz
$ rm nerdctl-full-1.7.2-linux-amd64.tar.gz SHA256SUMS
$ sudo mkdir -p /etc/systemd/system/user@.service.d
$ sudo tee /etc/systemd/system/user@.service.d/delegate.conf <<EOF
[Service]
Delegate=cpu cpuset io memory pids
EOF
$ sudo systemctl daemon-reload
$ containerd-rootless-setuptool.sh install
$ containerd-rootless-setuptool.sh install-buildkit
$ nerdctl run -d -p 5000:5000 --restart=always --name registry registry:2
$ nerdctl -n k8s.io build -t k8s-container-registry-1:5000/my-node-app:latest /home/administrator/my-node-app/
$ nerdctl -n k8s.io push --insecure-registry k8s-container-registry-1:5000/my-node-app

Completed on k8s-worker-node-1 server as a normal user and then as root for it to work:

$ curl -LO "https://github.com/containerd/nerdctl/releases/download/v1.7.2/nerdctl-1.7.2-linux-amd64.tar.gz"
$ curl -LO "https://github.com/containerd/nerdctl/releases/download/v1.7.2/SHA256SUMS"
$ sha256sum -c <(grep 'nerdctl-1.7.2-linux-amd64.tar.gz' 'SHA256SUMS')
$ sudo tar -C /usr/local/bin -xzf nerdctl-1.7.2-linux-amd64.tar.gz
$ rm nerdctl-1.7.2-linux-amd64.tar.gz SHA256SUMS
$ nerdctl -n k8s.io pull --insecure-registry k8s-container-registry-1:5000/my-node-app
FATA[0000] rootless containerd not running? (hint: use `containerd-rootless-setuptool.sh install` to start rootless containerd): stat /run/user/1000/containerd-rootless: no such file or directory
$ sudo mkdir -p /etc/systemd/system/user@.service.d
$ sudo tee /etc/systemd/system/user@.service.d/delegate.conf <<EOF
[Service]
Delegate=cpu cpuset io memory pids
EOF
$ sudo systemctl daemon-reload
$ curl -LO "https://github.com/rootless-containers/rootlesskit/releases/download/v2.0.0/rootlesskit-x86_64.tar.gz"
$ curl -LO "https://github.com/rootless-containers/rootlesskit/releases/download/v2.0.0/SHA256SUMS"
$ sha256sum -c <(grep 'rootlesskit-x86_64.tar.gz' 'SHA256SUMS')
$ sudo tar -C /usr/local/bin -xzf rootlesskit-x86_64.tar.gz
$ rm rootlesskit-x86_64.tar.gz SHA256SUMS
$ containerd-rootless-setuptool.sh install

Does not work as a normal user:
$ nerdctl -n k8s.io pull --insecure-registry k8s-container-registry-1:5000/my-node-app
$ kubectl describe pod my-node-app-7587fd66bb-shjb4
Events:
  Type     Reason             Age               From               Message
  ----     ------             ----              ----               -------
  Normal   Scheduled          11s               default-scheduler  Successfully assigned default/my-node-app-7587fd66bb-shjb4 to k8s-worker-node-1
  Warning  ErrImageNeverPull  9s (x3 over 11s)  kubelet            Container image "k8s-container-registry-1:5000/my-node-app:latest" is not present with pull policy of Never
  Warning  Failed             9s (x3 over 11s)  kubelet            Error: ErrImageNeverPull

Works as root user, how can we make the regular user work instead?:
# sudo /usr/local/bin/nerdctl -n k8s.io pull --insecure-registry k8s-container-registry-1:5000/my-node-app
$ kubectl describe pod my-node-app-7587fd66bb-f5vb4
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  8m47s  default-scheduler  Successfully assigned default/my-node-app-7587fd66bb-f5vb4 to k8s-worker-node-1
  Normal  Pulled     8m46s  kubelet            Container image "k8s-container-registry-1:5000/my-node-app:latest" already present on machine
  Normal  Created    8m46s  kubelet            Created container my-node-app
  Normal  Started    8m46s  kubelet            Started container my-node-app

Not sure what to make of it because as a regular user it only shows the image I've pushed and I assume kubernetes can't see nerdctl images of regular users?

$ nerdctl -n k8s.io images
REPOSITORY                              TAG       IMAGE ID        CREATED           PLATFORM       SIZE       BLOB SIZE
k8s-container-registry-1:5000/my-node-app    latest    478c1ec7c1c8    17 minutes ago    linux/amd64    1.1 GiB    382.7 MiB
<none>                                  <none>    478c1ec7c1c8    17 minutes ago    linux/amd64    1.1 GiB    382.7 MiB

However, if I check it under root I get many more images and for some reason kubernetes can see this image

# nerdctl -n k8s.io images
REPOSITORY                                   TAG        IMAGE ID        CREATED          PLATFORM       SIZE         BLOB SIZE
calico/cni                                   v3.27.0    2d38b2f9a14c    7 hours ago      linux/amd64    201.4 MiB    90.3 MiB
calico/cni                                   <none>     2d38b2f9a14c    7 hours ago      linux/amd64    201.4 MiB    90.3 MiB
calico/csi                                   v3.27.0    15a0c4813cea    7 hours ago      linux/amd64    16.6 MiB     8.3 MiB
calico/csi                                   <none>     15a0c4813cea    7 hours ago      linux/amd64    16.6 MiB     8.3 MiB
calico/node-driver-registrar                 v3.27.0    c6d36f4809be    7 hours ago      linux/amd64    21.5 MiB     10.6 MiB
calico/node-driver-registrar                 <none>     c6d36f4809be    7 hours ago      linux/amd64    21.5 MiB     10.6 MiB
calico/node                                  v3.27.0    58cd8f6547b4    7 hours ago      linux/amd64    328.2 MiB    111.3 MiB
calico/node                                  <none>     58cd8f6547b4    7 hours ago      linux/amd64    328.2 MiB    111.3 MiB
calico/pod2daemon-flexvol                    v3.27.0    d2cab59bf301    7 hours ago      linux/amd64    14.7 MiB     7.2 MiB
calico/pod2daemon-flexvol                    <none>     d2cab59bf301    7 hours ago      linux/amd64    14.7 MiB     7.2 MiB
k8s-container-registry-1:5000/my-node-app    latest     478c1ec7c1c8    8 minutes ago    linux/amd64    1.1 GiB      382.7 MiB
registry.k8s.io/kube-proxy                   v1.29.1    70ca1eed9fdf    7 hours ago      linux/amd64    81.6 MiB     27.0 MiB
registry.k8s.io/kube-proxy                   <none>     70ca1eed9fdf    7 hours ago      linux/amd64    81.6 MiB     27.0 MiB
registry.k8s.io/pause                        3.9        7031c1b28338    7 hours ago      linux/amd64    728.0 KiB    314.0 KiB
registry.k8s.io/pause                        <none>     7031c1b28338    7 hours ago      linux/amd64    728.0 KiB    314.0 KiB
<none>                                       <none>     58cd8f6547b4    7 hours ago      linux/amd64    328.2 MiB    111.3 MiB
<none>                                       <none>     70ca1eed9fdf    7 hours ago      linux/amd64    81.6 MiB     27.0 MiB
<none>                                       <none>     d2cab59bf301    7 hours ago      linux/amd64    14.7 MiB     7.2 MiB
<none>                                       <none>     2d38b2f9a14c    7 hours ago      linux/amd64    201.4 MiB    90.3 MiB
<none>                                       <none>     478c1ec7c1c8    8 minutes ago    linux/amd64    1.1 GiB      382.7 MiB
<none>                                       <none>     15a0c4813cea    7 hours ago      linux/amd64    16.6 MiB     8.3 MiB
<none>                                       <none>     c6d36f4809be    7 hours ago      linux/amd64    21.5 MiB     10.6 MiB
<none>                                       <none>     7031c1b28338    7 hours ago      linux/amd64    728.0 KiB    314.0 KiB