Zot registry as a proxy for mirroring:

[kubebn]

I was wondering if I overlooked anything in this part of the setup.

cat patch

version: v1alpha1
debug: false
persist: true
machine:
    registries:
        mirrors:
            docker.io:
                endpoints:
                    - http://localhost:5000/docker
                overridePath: true
            gcr.io:
                endpoints:
                    - http://localhost:5000/gcr
                overridePath: true
            ghcr.io:
                endpoints:
                    - http://localhost:5000/ghcr
                overridePath: true
            nvcr.io:
                endpoints:
                    - http://localhost:5000/nvcr
                overridePath: true
            quay.io:
                endpoints:
                    - http://localhost:5000/quay
                overridePath: true
            registry.k8s.io:
                endpoints:
                    - http://localhost:5000/k8s
                overridePath: true
cluster:
    allowSchedulingOnControlPlanes: true

talosctl cluster create --config-patch=@patch
validating CIDR and reserving IPs
generating PKI and tokens
creating state directory in "/root/.talos/clusters/talos-default"
creating network talos-default
creating controlplane nodes
creating worker nodes
renamed talosconfig context "talos-default" -> "talos-default-5"
waiting for API
bootstrapping cluster
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: OK
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: OK
waiting for no diagnostics: OK
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: OK
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: OK
waiting for all k8s nodes to report ready: OK
waiting for kube-proxy to report ready: OK
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: OK

merging kubeconfig into "/root/.kube/config"
renamed cluster "talos-default" -> "talos-default-3"
renamed auth info "admin@talos-default" -> "admin@talos-default-3"
renamed context "admin@talos-default" -> "admin@talos-default-3"
PROVISIONER           docker
NAME                  talos-default
NETWORK NAME          talos-default
NETWORK CIDR          10.5.0.0/24
NETWORK GATEWAY       10.5.0.1
NETWORK MTU           1500
KUBERNETES ENDPOINT   https://127.0.0.1:41993

NODES:

NAME                            TYPE           IP         CPU    RAM      DISK
/talos-default-controlplane-1   controlplane   10.5.0.2   2.00   2.1 GB   -
/talos-default-worker-1         worker         10.5.0.3   2.00   2.1 GB   -

---
# Source: zot/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: zot
  namespace: kube-system
  labels:
    helm.sh/chart: zot-0.1.71
    app.kubernetes.io/name: zot
    app.kubernetes.io/instance: zot
    app.kubernetes.io/version: "v2.1.3"
    app.kubernetes.io/managed-by: Helm
---
# Source: zot/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: zot-config
  namespace: kube-system
data:
  # values taken from: https://github.com/71g3pf4c3/lomtik/blob/867de54d42e01f98906f03715b51dbb995b1ceef/packages/apps/flux-instance/resourceset.yaml#L69
  config.json: "{\n  \"storage\": { \"rootDirectory\": \"/var/lib/registry\" },\n  \"http\": { \"address\": \"0.0.0.0\", \"port\": \"5000\" },\n  \"log\": { \"level\": \"debug\" },\n  \"distSpecVersion\": \"1.0.1\",\n  \"extensions\": {\n    \"sync\": {\n      \"enable\": true,\n      \"registries\": [\n        {\n          \"urls\": [\"https://k8s.gcr.io\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/k8s-ghcr\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://docker.io\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/docker\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://quay.io\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/quay\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://ghcr.io\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/ghcr\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://gcr.io\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/gcr\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://registry.k8s.io\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/k8s\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://mcr.microsoft.com\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/mcr\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://docker.elastic.co\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/elastic\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        },\n        {\n          \"urls\": [\"https://nvcr.io\"],\n          \"content\": [\n            {\n              \"prefix\": \"**\", \n              \"destination\": \"/nvcr\"\n            }\n          ],\n          \"onDemand\": true,\n          \"tlsVerify\": true\n        }\n      ]\n    }\n  }\n}"
---
# Source: zot/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: zot
  namespace: kube-system
  labels:
    helm.sh/chart: zot-0.1.71
    app.kubernetes.io/name: zot
    app.kubernetes.io/instance: zot
    app.kubernetes.io/version: "v2.1.3"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 5000
      targetPort: 5000
      protocol: TCP
      name: zot
  selector:
    app.kubernetes.io/name: zot
    app.kubernetes.io/instance: zot
---
# Source: zot/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: zot
  namespace: kube-system
  labels:
    helm.sh/chart: zot-0.1.71
    app.kubernetes.io/name: zot
    app.kubernetes.io/instance: zot
    app.kubernetes.io/version: "v2.1.3"
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: zot
      app.kubernetes.io/instance: zot
  template:
    metadata:
      annotations:
        checksum/config: b88c9620f76afe66a33ce1dd246894d1ee5ed96421515d0ba8df454473978e5a
      labels:
        app.kubernetes.io/name: zot
        app.kubernetes.io/instance: zot
    spec:
      serviceAccountName: zot
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      securityContext:
        null
      containers:
        - name: zot
          securityContext:
            null
          image: "ghcr.io/project-zot/zot-linux-amd64:v2.1.2"
          imagePullPolicy: IfNotPresent
          env:
            null
          ports:
            - name: zot
              containerPort: 5000
              protocol: TCP
              hostPort: 5000
          volumeMounts:
            - name: data
              mountPath: /var/lib/registry
            - mountPath: '/etc/zot'
              name: zot-config
          livenessProbe:
            initialDelaySeconds: 5
            httpGet:
              path: /v2/
              port: 5000
              scheme: HTTP
          readinessProbe:
            initialDelaySeconds: 5
            httpGet:
              path: /v2/
              port: 5000
              scheme: HTTP
          startupProbe:
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3
            httpGet:
              path: /v2/
              port: 5000
              scheme: HTTP
          resources:
            null
      volumes:
        - name: data
          hostPath:
            path: /var/lib/registry
            type: DirectoryOrCreate
        - name: zot-config
          configMap:
            name: zot-config
      tolerations:
        - operator: Exists

---
stern 'zot-*'

zot-cnd7l zot {"level":"info","goroutine":1,"caller":"zotregistry.dev/zot/pkg/extensions/extension_ui.go:67","time":"2025-05-26T12:12:00.562855327Z","message":"skip enabling the ui route as the config prerequisites are not met"}
zot-cnd7l zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.2:34776","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":451,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:10.534008079Z","message":"HTTP API"}
zot-cnd7l zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.2:34778","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":453,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:10.534650716Z","message":"HTTP API"}
zot-vkw54 zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.3:50832","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":444,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:20.530406594Z","message":"HTTP API"}
zot-vkw54 zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.3:50846","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":446,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:20.531072381Z","message":"HTTP API"}
zot-cnd7l zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.2:59948","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":455,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:20.533864157Z","message":"HTTP API"}
zot-cnd7l zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.2:59964","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":439,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:20.534915275Z","message":"HTTP API"}
zot-vkw54 zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.3:44968","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":426,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:30.530771029Z","message":"HTTP API"}
zot-vkw54 zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.3:44978","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":448,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:30.531162194Z","message":"HTTP API"}
zot-cnd7l zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.2:55504","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":420,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:30.533997442Z","message":"HTTP API"}
zot-cnd7l zot {"level":"info","module":"http","component":"session","clientIP":"10.5.0.2:55514","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.33"]},"goroutine":441,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-05-26T12:12:30.53464937Z","message":"HTTP API"}

---

# Define images to test
IMAGES=(
    "docker.io/library/nginx:latest"
    "quay.io/curl/curl:8.13.0"
    "nvcr.io/nvidia/k8s-device-plugin:v0.17.1"
    "registry.k8s.io/pause:3.10"
    "gcr.io/etcd-development/etcd:v3.5.21"
    "ghcr.io/siderolabs/flannel:v0.26.7"
)

# Deploy each image in talos
for IMAGE in "${IMAGES[@]}"; do
    DEPLOYMENT_NAME=$(echo "$IMAGE" | tr '/:' '-')
    echo "Creating deployment for image: $IMAGE"
    kubectl create deployment "$DEPLOYMENT_NAME" --image="$IMAGE" --dry-run=client -o yaml | kubectl apply -f -
    if [[ $? -ne 0 ]]; then
        echo "Failed to create deployment for image: $IMAGE"
        exit 1
    fi
done

---

After deployments are scheduled, I still can't see images in the catalog:

curl http://10.5.0.2:5000/v2/_catalog
{"repositories":[]}#

curl http://10.5.0.3:5000/v2/_catalog
{"repositories":[]}#

[smira]

You can check the logs of CRI (talosctl logs cri) to see the image pull process specifics.

Please keep in mind in your setup there's a fallback, so after localhost:5000 CRI would try default upstream.

[kubebn]

You can check the logs of CRI (talosctl logs cri) to see the image pull process specifics.

Please keep in mind in your setup there's a fallback, so after localhost:5000 CRI would try default upstream.

10.5.0.3: {"level":"info","msg":"StartContainer for \"13a57acdd85a728482f08033490672feedd177c389be5220231e448bb63d6dfb\"","time":"2025-05-27T09:39:58.520160943Z"}
10.5.0.3: {"address":"unix:///run/containerd/s/a7aab6f25c15b339ef026f64563cf6ca815f7edfb51b431875a04419bc027ac1","level":"info","msg":"connecting to shim 13a57acdd85a728482f08033490672feedd177c389be5220231e448bb63d6dfb","namespace":"k8s.io","protocol":"ttrpc","time":"2025-05-27T09:39:58.523868649Z","version":3}
10.5.0.3: {"level":"info","msg":"StartContainer for \"13a57acdd85a728482f08033490672feedd177c389be5220231e448bb63d6dfb\" returns successfully","time":"2025-05-27T09:39:58.571116385Z"}
10.5.0.3: {"level":"info","msg":"ImageCreate event name:\"docker.io/library/nginx:latest\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}","time":"2025-05-27T09:39:59.154118127Z"}
10.5.0.3: {"level":"info","msg":"stop pulling image docker.io/library/nginx:latest: active requests=0, bytes read=72412935","time":"2025-05-27T09:39:59.154209417Z"}
10.5.0.3: {"level":"info","msg":"ImageCreate event name:\"sha256:be69f2940aaf64fdf50c9c99420cbd57e10ee655ec7204df1c407e9af63d0cc1\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}","time":"2025-05-27T09:39:59.155355432Z"}
10.5.0.3: {"level":"info","msg":"ImageCreate event name:\"docker.io/library/nginx@sha256:fb39280b7b9eba5727c884a3c7810002e69e8f961cc373b89c92f14961d903a0\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}","time":"2025-05-27T09:39:59.162487795Z"}
10.5.0.3: {"level":"info","msg":"Pulled image \"docker.io/library/nginx:latest\" with image id \"sha256:be69f2940aaf64fdf50c9c99420cbd57e10ee655ec7204df1c407e9af63d0cc1\", repo tag \"docker.io/library/nginx:latest\", repo digest \"docker.io/library/nginx@sha256:fb39280b7b9eba5727c884a3c7810002e69e8f961cc373b89c92f14961d903a0\", size \"72402122\" in 4.312245551s","time":"2025-05-27T09:39:59.163800325Z"}
10.5.0.3: {"level":"info","msg":"PullImage \"docker.io/library/nginx:latest\" returns image reference \"sha256:be69f2940aaf64fdf50c9c99420cbd57e10ee655ec7204df1c407e9af63d0cc1\"","time":"2025-05-27T09:39:59.163817415Z"}
10.5.0.3: {"level":"info","msg":"CreateContainer within sandbox \"878493cf4590ac8a0e540b6e080221213dd4dfd96489bb855ad0b0720be0c4a2\" for container \u0026ContainerMetadata{Name:nginx,Attempt:0,}","time":"2025-05-27T09:39:59.173227182Z"}
10.5.0.3: {"level":"info","msg":"Container 5557ef41883c959dc86562a49e491e964084afb46e8852bbaa1554e25f770635: CDI devices from CRI Config.CDIDevices: []","time":"2025-05-27T09:39:59.195336006Z"}
10.5.0.3: {"level":"info","msg":"CreateContainer within sandbox \"878493cf4590ac8a0e540b6e080221213dd4dfd96489bb855ad0b0720be0c4a2\" for \u0026ContainerMetadata{Name:nginx,Attempt:0,} returns container id \"5557ef41883c959dc86562a49e491e964084afb46e8852bbaa1554e25f770635\"","time":"2025-05-27T09:39:59.307120259Z"}
10.5.0.3: {"level":"info","msg":"StartContainer for \"5557ef41883c959dc86562a49e491e964084afb46e8852bbaa1554e25f770635\"","time":"2025-05-27T09:39:59.307512793Z"}
10.5.0.3: {"address":"unix:///run/containerd/s/becd88d7edf71224b536729ce3630242681a6e184b8abeca8daf8153eef8dfc7","level":"info","msg":"connecting to shim 5557ef41883c959dc86562a49e491e964084afb46e8852bbaa1554e25f770635","namespace":"k8s.io","protocol":"ttrpc","time":"2025-05-27T09:39:59.311499207Z","version":3}

Does not seem like it uses localhost anywhere in the logs. In the mc for the nodes I see that mirrors is set though:

    registries:
        # Specifies mirror configuration for each registry host namespace.
        mirrors:
            docker.io:
                # List of endpoints (URLs) for registry mirrors to use.
                endpoints:
                    - http://localhost:5000/docker
                overridePath: true # Use the exact path specified for the endpoint (don't append /v2/).
            gcr.io:
                # List of endpoints (URLs) for registry mirrors to use.
                endpoints:
                    - http://localhost:5000/gcr
                overridePath: true # Use the exact path specified for the endpoint (don't append /v2/).
            ghcr.io:
                # List of endpoints (URLs) for registry mirrors to use.
                endpoints:
                    - http://localhost:5000/ghcr
                overridePath: true # Use the exact path specified for the endpoint (don't append /v2/).
            nvcr.io:
                # List of endpoints (URLs) for registry mirrors to use.
                endpoints:
                    - http://localhost:5000/nvcr
                overridePath: true # Use the exact path specified for the endpoint (don't append /v2/).
            quay.io:
                # List of endpoints (URLs) for registry mirrors to use.
                endpoints:
                    - http://localhost:5000/quay
                overridePath: true # Use the exact path specified for the endpoint (don't append /v2/).
            registry.k8s.io:
                # List of endpoints (URLs) for registry mirrors to use.
                endpoints:
                    - http://localhost:5000/k8s
                overridePath: true # Use the exact path specified for the endpoint (don't append /v2/).

[smira]

There is no magic here, in the end Talos generates hosts configuration for containerd, and it just uses that.

You should see the attempt to pull as long as the image is not cached.