Trying to figure out why I get access denied

[ghita]

I run on WSL2 (Win11)
ramalama --debug convert hf://qwen/qwen3-4B oci://localhost:5000/qwen3:4b

The files get downloaded and at some point I see in the logs:
`2025-06-27 23:10:46 - DEBUG - Command finished with return code: 0
2025-06-27 23:10:46 - WARNING - ⚠️ Failed to read model '/home/gheor/.local/share/ramalama/store/huggingface/qwen/qwen3-4B/blobs/sha256-2de5ee7eee214bb55ea33ec7505c5838a7adf7f6': [Errno 2] No such file or directory: '/home/gheor/.local/share/ramalama/store/huggingface/qwen/qwen3-4B/blobs/sha256-2de5ee7eee214bb55ea33ec7505c5838a7adf7f6'
Converting /home/gheor/.local/share/ramalama/store to /home/gheor/.local/share/ramalama/store ...
2025-06-27 23:10:46 - DEBUG - run_cmd: docker manifest rm localhost:5000/qwen3:4b
2025-06-27 23:10:46 - DEBUG - Working directory: None
2025-06-27 23:10:46 - DEBUG - Ignore stderr: True
2025-06-27 23:10:46 - DEBUG - Ignore all: False
Building localhost:5000/qwen3:4b ...
2025-06-27 23:10:47 - DEBUG - run_cmd: docker build --no-cache --network=none -q -f /tmp/RamaLama_Containerfile_kksctac4 /home/gheor/.local/share/ramalama/store/huggingface/qwen/qwen3-4B/blobs
2025-06-27 23:10:47 - DEBUG - Working directory: None
2025-06-27 23:10:47 - DEBUG - Ignore stderr: False
2025-06-27 23:10:47 - DEBUG - Ignore all: False
2025-06-27 23:13:17 - DEBUG - Command finished with return code: 0
2025-06-27 23:13:17 - DEBUG - run_cmd: docker manifest create localhost:5000/qwen3:4b sha256:19ab94ecd645e26283652adad32906617e5012077fc850ad495db476ef1663dd
2025-06-27 23:13:17 - DEBUG - Working directory: None
2025-06-27 23:13:17 - DEBUG - Ignore stderr: False
2025-06-27 23:13:17 - DEBUG - Ignore all: False
errors:
denied: requested access to the resource is denied
unauthorized: authentication required

Failed to create manifest for OCI localhost:5000/qwen3:4b : Command '['docker', 'manifest', 'create', 'localhost:5000/qwen3:4b', 'sha256:19ab94ecd645e26283652adad32906617e5012077fc850ad495db476ef1663dd']' returned non-zero exit status 1.
Tagging build instead`

I have an denied: requested access to the resource is denied but is not clear to me what reports that.

[rhatdan]

Any chance you can try this with Podman?

[ghita]

I am not sure if podman can use HW acceleration inside WSL2. I had to install "NVIDIA Container Toolkit" inside WSL to be able to use acceleration support.
Not that familiar with podman, and if I can run it along with my Windows Docker Desktop without issues.
Is there a way to update some logging in ramalama to see from where we get "denied: requested access to the resource is denied" ? Or maybe it's quite clear that is caused by "'docker', 'manifest', 'create'".

[rhatdan]

RamaLama is just running a docker command. You can see the docker command it is running by doing

ramalama --debug or ramalama --dryrun

Most people that are doing WSL with this now are entering the VM and running the RamaLama command in the VM.
But I would like to know what is going wrong.

It very well could be something to do with the way that docker is volume mounting into the VM.

[ghita]

Looked more into it, indeed _create_manifest() from oci.py fails with that authorization error.
Unfortunately I am not that into oci and what this manifest represents but a quick chatbot search says it could be caused either by my local registry deployment or docker client. I mention that a simple image push to my registry works, so I am not sure what's going on. Initially I had some issues with the way registry was configured. Might I have some intermediate state in ramalama from previous failure I need to clean locally ?

[rhatdan]

No idea, but as I said in this case all RamaLama is doing is docker build, docker artifact create, docker push. Unless RamaLama is not passing an option to docker then this seems to be an issue between docker and your registry.

Running all of the commands with --debug should show you all of the docker commands executed.

[ericcurtin]

I'm not sure how we tackle this right now, this is a safetensors repo, we haven't really handled safetensors yet. But we should see how these fit into the newly defined CNCF standard for OCI artifacts.

[ghita]

I have conversion issue even with the example given in the documentation "ramalama convert ollama://tinyllama:latest oci://localhost:5000/tiny:latest" I guess that one does not use safetensors ? I don't really know why that steps fails with auth error. Does the tag for my local doker repo has anything to do with that auth error or it has nothing to do with it as an actual "conversation" with localhost:5000 won't happen till we push to the target repo ?

[rhatdan]

Have you done a convert with ramalama --debug convert ...

To show us which step this happens with?

[ghita]

It's same as in the report
`ramalama --debug convert ollama://tinyllama:latest oci://localhost:5000/tiny:latest
2025-07-01 20:51:32 - DEBUG - run_cmd: nvidia-smi
2025-07-01 20:51:32 - DEBUG - Working directory: None
2025-07-01 20:51:32 - DEBUG - Ignore stderr: False
2025-07-01 20:51:32 - DEBUG - Ignore all: False
2025-07-01 20:51:32 - DEBUG - Command finished with return code: 0
2025-07-01 20:51:32 - DEBUG - run_cmd: docker inspect quay.io/ramalama/cuda:0.9
2025-07-01 20:51:32 - DEBUG - Working directory: None
2025-07-01 20:51:32 - DEBUG - Ignore stderr: False
2025-07-01 20:51:32 - DEBUG - Ignore all: True
2025-07-01 20:51:32 - DEBUG - Command finished with return code: 0
2025-07-01 20:51:32 - DEBUG - run_cmd: nvidia-smi
2025-07-01 20:51:32 - DEBUG - Working directory: None
2025-07-01 20:51:32 - DEBUG - Ignore stderr: False
2025-07-01 20:51:32 - DEBUG - Ignore all: False
2025-07-01 20:51:32 - DEBUG - Command finished with return code: 0
2025-07-01 20:51:32 - DEBUG - run_cmd: docker inspect quay.io/ramalama/cuda:0.9
2025-07-01 20:51:32 - DEBUG - Working directory: None
2025-07-01 20:51:32 - DEBUG - Ignore stderr: False
2025-07-01 20:51:32 - DEBUG - Ignore all: True
2025-07-01 20:51:32 - DEBUG - Command finished with return code: 0
Converting /home/gheor/.local/share/ramalama/store to /home/gheor/.local/share/ramalama/store ...
2025-07-01 20:51:32 - DEBUG - run_cmd: docker manifest rm localhost:5000/tiny:latest
2025-07-01 20:51:32 - DEBUG - Working directory: None
2025-07-01 20:51:32 - DEBUG - Ignore stderr: True
2025-07-01 20:51:32 - DEBUG - Ignore all: False
Building localhost:5000/tiny:latest ...
2025-07-01 20:51:32 - DEBUG - run_cmd: docker build --no-cache --network=none -q -f /tmp/RamaLama_Containerfile_v7g6mbva /home/gheor/.local/share/ramalama/store/ollama/tinyllama/tinyllama/blobs
2025-07-01 20:51:32 - DEBUG - Working directory: None
2025-07-01 20:51:32 - DEBUG - Ignore stderr: False
2025-07-01 20:51:32 - DEBUG - Ignore all: False
2025-07-01 20:51:50 - DEBUG - Command finished with return code: 0
2025-07-01 20:51:50 - DEBUG - run_cmd: docker manifest create localhost:5000/tiny:latest sha256:b21820738ce3395011aea4539574fd501165f2a1074921987b3d53920d14cc76
2025-07-01 20:51:50 - DEBUG - Working directory: None
2025-07-01 20:51:50 - DEBUG - Ignore stderr: False
2025-07-01 20:51:50 - DEBUG - Ignore all: False
errors:
denied: requested access to the resource is denied
unauthorized: authentication required

Failed to create manifest for OCI localhost:5000/tiny:latest : Command '['docker', 'manifest', 'create', 'localhost:5000/tiny:latest', 'sha256:b21820738ce3395011aea4539574fd501165f2a1074921987b3d53920d14cc76']' returned non-zero exit status 1.
Tagging build instead
2025-07-01 20:51:52 - DEBUG - run_cmd: docker tag sha256:b21820738ce3395011aea4539574fd501165f2a1074921987b3d53920d14cc76 localhost:5000/tiny:latest
2025-07-01 20:51:52 - DEBUG - Working directory: None
2025-07-01 20:51:52 - DEBUG - Ignore stderr: False
2025-07-01 20:51:52 - DEBUG - Ignore all: False
2025-07-01 20:51:52 - DEBUG - Command finished with return code: 0`

[rhatdan]

docker manifest create localhost:5000/tiny:latest sha256:b21820738ce3395011aea4539574fd501165f2a1074921987b3d53920d14cc76

If you execute this at the command line, does it report the same error?

[ghita]

I could try to install podman on my Win machine also, but not sure if there would be any conflicts having multiple container engines. (could also get rid of docker maybe at some point as I have issues/restarts triggered by WSL integration problems all the time ;-))

[rhatdan]

I have a feeling that this is a general issue with docker manifest create.

If you tried to create a manifest using an existing image, would it work?

[ghita]

Not sure what I could try, if you give me a specific example. In the meantime I installed podman and seems to be available from WSL2 container. But the project keeps wanting to connect to docker still. Haven't looked to much on the detection logic but I had the impression it picks podman first...
`ramalama info

Error: Command '['docker', 'info', '--format', 'json']' returned non-zero exit status 1.`

[rhatdan]

You can change engine to podman within ramalama.conf file.

[ghita]

yep, with podman the convert works as expected. Must be something about docker support I guess.