Image Cache Issue with Talos 1.9.4

[Martin-Lierschof]

We have had a discussion about this on slack before here, i was asked to post it here:
https://taloscommunity.slack.com/archives/CG25RPZNE/p1741710681879299

Summary

I encountered an issue with Talos 1.9.4 related to the image cache mechanism. While the OCI folder creation and ISO build process succeed, and Talos does not report any errors during launch, the image cache does not work as expected. The logs indicate missing images, and the directory structure appears to be incorrect after copying to the disk.

Observations

Image Cache Configuration

After booting the node, the image-cache metadata reports as follows:

node: xxxx-cp-1
metadata:
    namespace: cri
    type: ImageCacheConfigs.cri.talos.dev
    id: image-cache
    version: 4
    owner: cri.ImageCacheConfigController
    phase: running
    created: 2025-03-11T15:49:57Z
    updated: 2025-03-11T15:49:58Z
spec:
    status: ready
    copyStatus: ready
    roots:
        - /system/imagecache/disk
        - /system/imagecache/iso/imagecache

However, the logs show errors indicating missing image references:

ERROR failed to handle request {"error": "2 errors occurred:\n\t* open /system/imagecache/disk/manifests/registry.k8s.io/pause/reference/3.10: no such file or directory\n\t* open /system/imagecache/iso/imagecache/manifests/registry.k8s.io/pause/reference/3.10: no such file or directory\n\n"}

Structure of the Image Cache Folders

The structure of the /system/imagecache/iso/imagecache/ folder appears incorrect. The expected hierarchy is not maintained, leading to a flat directory structure with backslashes () in paths:

talosctl ls /system/imagecache/iso/imagecache

manifests\registry.k8s.io
manifests\registry.k8s.io\coredns
manifests\registry.k8s.io\coredns\coredns
manifests\registry.k8s.io\coredns\coredns\digest
manifests\registry.k8s.io\coredns\coredns\digest\sha256-2324f485c8db937628a18c293d946327f3a7229b9f77213e8f2256f0b616a4ee

Behavior of talosctl images pull

Even after manually pulling an image:

talosctl images pull ghcr.io/loafoe/go-hello-world

The image appears in the list, but the cache directories (/system/imagecache/disk/ and /system/imagecache/iso/imagecache/) do not reflect any changes. The expected sha256 blobs are also missing.

Pre-ISO Image Cache Structure

Before building the Talos ISO, the local cache folder follows the standard OCI layout:

image-layer-cache

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          12/03/2025    07:51        5689289 sha256-01b51d...
-a---          12/03/2025    07:51             75 sha256-02c91f...
-a---          12/03/2025    07:51            101 sha256-0b2215...

Whereas the image cache folder contains:

image-cache

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          12/03/2025    07:52                blobs
-a---          12/03/2025    07:52            405 index.json
-a---          12/03/2025    07:52             37 oci-layout

Within image-cache/blobs/sha256/, the blobs are properly stored:

image-cache\blobs\sha256

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          12/03/2025    07:52            413 83247d2...
-a---          12/03/2025    07:52      999855775 c2f54a7...
-a---          12/03/2025    07:52            246 cce2fc0...

Potential Causes

1. ISO File System Differences: Could this be related to handling of ISOs with vfat and iso9660? There are comments in Talos code suggesting certain volume names might influence the structure.
2. Incorrect Path Handling in Talos: The transformation of forward slashes (/) to backslashes () might indicate improper path handling when copying images into the ISO.
3. Failure to Persist Image Blobs: Even though Talos reports successful image pulls, the cache does not update, which may suggest an issue with persistence or copying mechanisms.

Next Steps

• Investigate how Talos handles path conversions when copying the image cache to the ISO.
• Verify whether file system format (e.g., vfat vs. iso9660) affects the directory structure inside the Talos environment.
• Check logs for any indication of why talosctl images pull does not modify the expected cache locations.
• Compare behavior on different storage backends (e.g., RAM disk vs. persistent disk) to determine if caching failures are storage-related.

Any insights or suggestions are welcome!

[smira]

Are you using Windows talosctl to create the image cache?

[Martin-Lierschof]

Yes its windows

[smira]

Ok, I guess you're hitting path delimiter issues here. I'll move this to an issue, but in the meantime it might be easier to use Linux talosctl under WSL.

[Martin-Lierschof]

Sadly in our case this is not possible. Can we help somehow?

[smira]

You can build the image cache anywhere literally and move the OCI blob. We will get the bug fixed (#10558), but it might take some time.

[Martin-Lierschof]

So as a workaround we are running this in podman (c# code):
This launches an alpine linux and download and runs talosctl in there.
Here we also needed to add the rootCA for our private internal repos.

string imagesArg = string.Join(" ", allImages.Select(img => $"--images='{img}'"));
string command= $"podman run --rm -it " +
$"-v \"{ImageCacheDirectory}:/mnt/imagecache\" " +
$"-v \"{ImageCacheLayerDirectory}:/mnt/imagecachelayer\" " +
$"-v \"{CertExportPath}:/usr/local/share/ca-certificates/rootCA.crt\" " +
"alpine sh -c \"" +
"apk add --no-cache ca-certificates curl && " +
"update-ca-certificates && " +
"curl -Lo /usr/local/bin/talosctl https://github.com/siderolabs/talos/releases/latest/download/talosctl-linux-amd64 && " +
"chmod +x /usr/local/bin/talosctl && " +
 $"talosctl images cache-create --force --insecure --image-layer-cache-path /mnt/imagecachelayer --image-cache-path /mnt/imagecache/cache.oci {imagesArg}\"";