[SURE-8882] extend our testing to ssh helm chart downloads with keys

[kkaempf]

Acceptance Criteria

• write a test for Helm chart downloads via SSH as in https://github.com/rancher/fleet/blob/43cf0a41330c57c3d1b853e00ab66ab2c1899d6a/internal/bundlereader/loaddirectory.go#L242C2-L270C3

SURE-8882

Issue description:

The customer upgraded from Rancher 2.8.2 to Rancher 2.8.5 and some of their upstream fleet jobs are getting this error:

time=2024-07-30 15:16:18.000000 level=fatal msg="error downloading 'ssh://git@github.com/xxxxx/fleet-platform.git?sshkey=redacted': /usr/bin/git exited with 128: Cloning into '/tmp/getter624252719/temp'...\nNo user exists for uid 1000\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"

Troubleshooting steps:

The customer  tried changing the credentials and still get the same error.
They are able to clone the repository locally using the same credentials supplied to Fleet. This also happens on most of the configured repositories, not just one or two git repos.
They are able to exec into the gitjob pod and manually clone the repo with success.
Checked from inside the GitJob pod:

 > kubectl exec -n cattle-fleet-system gitjob-7889c69f49-5kq8r -it -- cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
sshd:x:499:486:SSH daemon:/var/lib/sshd:/usr/sbin/nologin
gitjob:x:1000:1000::/home/gitjob:/bin/bash

I reviewed two of their GitRepo manifests (working/not-working) and they are literally pointing to the same repo, the only difference is the path used.
The customer dowgraded from 0.9.5 to 0.9.0 and the problem repos started to sync again

Repro steps:

unable to repro in-house

Workaround:

Is a workaround available and implemented? yes/no
What is the workaround: Downgrade fleet

Actual behavior:

After upgrade, some gitrepos fail with error:

time=2024-07-30 15:16:18.000000 level=fatal msg="error downloading 'ssh://git@github.com/xxxxx/fleet-platform.git?sshkey=redacted': /usr/bin/git exited with 128: Cloning into '/tmp/getter624252719/temp'...\nNo user exists for uid 1000\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"

Expected behavior:
All gitrepos continue to sync with no error

Files, logs, traces:
 

Additional notes:

[cienijr]

We're facing the same issue here. SSH doesn't work at all when pulling helm charts.
It might be related to 326ad93, a quick Google search suggests that OpenSSH depends on the entry existing at /etc/passwd for some obscure reason.

Amusingly, using SSH for the GitRepo itself does work (the initcontainer runs fleet gitcloner and exits with success). It only seems to be an issue when running fleet apply.

We're using Rancher 2.9.3 with fleet 0.10.4

[manno]

We're facing the same issue here. SSH doesn't work at all when pulling helm charts. It might be related to 326ad93, a quick Google search suggests that OpenSSH depends on the entry existing at /etc/passwd for some obscure reason.

Amusingly, using SSH for the GitRepo itself does work (the initcontainer runs fleet gitcloner and exits with success). It only seems to be an issue when running fleet apply.

If downloading a helm chart via ssh fails, that would be a separate issue. The fleet apply CLI uses go-getter to download charts.
Cloning the git repository is done by fleet gitcloner and it uses go-git.