[SURE-10858] Fleet showing "Unauthorized" transiently on GitRepos #4252
Description
SURE-10858
Issue description:
Fleet is showing an "Unauthorized" error transiently on customer GitRepos in their nonprod environment. They use both git auth (Github) and helm auth (Artifactory) on their Repo.
Most times it throws the error in the screenshot but sometimes it looks all green/healthy. They've just been continually clicking "Force Update" and it eventually will sync their changes even with the error temporarily.
For reference, they've checked their GitRepo and artifactory credentials are correct and identical to their prod environment (which is not having this issue), in addition to their proxy config.
We aren't even sure what component is "Unauthorized" as of this moment.
Business impact:
Unable to consistently push changes in nonprod.
Troubleshooting steps:
We have enabled debug logging for the fleet controller, but since the job is before the actual bundle deployment there are no pertinent logs.
We've looked in logs for fleet-agent on downstream clusters as well. My understanding is that this job is responsible for cloning the git repo, and since it fails it doesn't even get far enough to spit out related logs in the downstream cluster. Hence the fleet-agent logs are not helpful.
I've attempted to recreate the problem but I'm not able to get the same message. Incorrect credentials don't show "Unauthorized" nor do they work transiently.
Repro steps:
unable to reproduce
Workaround:
Is a workaround available and implemented? no
Files, logs, traces:
See issue.