New job for GitRepo is created and terminated every 3rd second #2853
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
We have a Rancher installation (version 2.9.1), this problem likely started after upgrading from 2.8.x.
We have 3 GitRepos, but only one of them are experiencing this problem. All point to the same Git repository in BitBucket but with different paths. We run on EKS 1.28 currently but plan on upgrading to EKS 1.29 soon.
For one of these GitRepos a job/pod is created roughly every 3 seconds and then it is terminated (usually), but sometimes they get stuck and we run out of IP-addresses in the subnet. The other GitRepos only see new jobs occasionally or when changes are done to the backing Git repository.
The GitRepo with problem also has this warning/error which we don't understand why it is there:
User "system:serviceaccount:cattle-fleet-system:fleet-controller" cannot create resource "gitjobs" in API group "gitjob.cattle.io" in the namespace ""
Expected Behavior
Pods are not created every 3rd second.
Steps To Reproduce
No response
Environment
- Architecture: amd64
- Fleet Version: v0.10.1
- Cluster:
- Provider: EKS
- Options: 4 nodes, master node running fleet-controller is c6i.12xlarge to accommodate the number of clusters. Ingress Nginx, AWS LB.
- Kubernetes Version: 1.28Logs
stream logs failed container "fleet" in pod "<gitjob>-f0e21-g5cjr" is waiting to start: PodInitializing for <namespace>/<gitjob>-f0e21-g5cjr (fleet)
gitcloner-initializer time="2024-09-16T14:03:58Z" level=warning msg="signal received: \"terminated\", canceling context..."
Stream closed EOF for <namespace>/<gitjob>-f0e21-g5cjr (gitcloner-initializer)Anything else?
We see a lot of logs like this even though no changes are made to the backing Git repo in Bitbucket.
{"level":"info","ts":"2024-09-16T14:07:30Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"<namespace>","name":"cluster-8cf77d5971e8"}