Wait container tries to pull Linux image on Windows k8s node

[ilbatin]

Hello!
I’ve encountered an issue when trying to use Windows nodes for running Windows images in Argo Workflows.

The problem is that the wait container attempts to pull quay.io/argoproj/argoexec:v3.5.5 on a Windows machine, even though I specified the node selector kubernetes.io/os=windows.

The documentation says that The Argo server and the workflow controller currently only run on Linux. The workflow executor however also runs on Windows nodes, meaning you can use Windows containers inside your workflows!.
But how exactly do you achieve this? The argoexec image is specified in the controller configuration. Do I need to run two controllers—one for Linux and another for Windows?

Additional debug info:

Workflow config:

spec:
  templates:
    - name: windows-workflows
      inputs: {}
      outputs: {}
      nodeSelector:
        kubernetes.io/os: windows
      metadata: {}
      container:
        name: windows
        image: >-
          docker-registry/windows-workflows:0.0.1 # our custom image
        command:
          - powershell.exe
          - '-command'
          - echo test
        resources: {}

Error message:

Back-off pulling image "quay.io/argoproj/argoexec:v3.5.5"
Pulling image "quay.io/argoproj/argoexec:v3.5.5"
Error: ErrImagePull

K8S GKE cluster version: 1.30.5-gke.1443001
My workflow controller version is workflow-controller:v3.6.0

[Joibel]

The image quay.io/argoproj/argoexec:v3.5.5 is correct. It is a multi-platform image:

$ docker manifest inspect quay.io/argoproj/argoexec:v3.5.5
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2846,
         "digest": "sha256:9d783aa165850ef3c5e47c8dbe13cf96bf5008497460b4d1ed68e5a4c510b39e",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2846,
         "digest": "sha256:3ce0381dd63e7800befadaa8f8b8f20b3e516881db889566e0b00a266f92771a",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1995,
         "digest": "sha256:49c5fe6231477a44b61ac0e67600b4896237ad4ca815fd43a9d2c0b16a0ca1c8",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.20348.2322"
         }
      }
   ]
}

Image pull errors are not related to platform, this implies your windows node cannot pull the image at all. Docker will happily pull images for the wrong platform but then fail to run them.

[ilbatin]

Hey, I installed v3.6.0 and it did not help.
But I was able to manually pull the image from the Windows node with ctr image pull quay.io/argoproj/argoexec:v3.6.0. So you are right, it's possible to pull it on Windows.

Still, crictl fails to pull the image. I guess the problem is in k8s, for some reason it can't pull image.

[ilbatin]

@Joibel do you know by any chance if the fact that the image can be pulled manually from the Windows node via ctr (containerd ctl) means that our issue is k8s related?

[Joibel]

It feels likely.

Ignoring workflows completely:
Can you use any other multiplatform image inside your k8s cluster?
Can you use an image from GHCR?
Can you use a multiplatform image from GHCR?

[ilbatin]

The problem was in the difference in kernel version:

docker inspect quay.io/argoproj/argoexec:v3.6.0
"OsVersion": "10.0.20348.2849"

And my Windows node had kernelVersion: 10.0.17763.5329

The solution for the future generations is to downgrade argo or update windows

[Joibel]

Ah, nice find. Thanks for reporting back.

We could do better at documenting the exact versions for windows - I had forgotten that aspect of windows containers.