Benchmark Tests

[manno]

We want to measure how Fleet performs on a given system of clusters, for a set of tests.

For a future benchmark suite, create the first test. This can be a script or executable, a Go benchmark test, etc.
The test runs against a cluster and collects several measures.

It should create 5 GitRepo resources, which contain 2 bundles each. The bundles should be targeted at 5 downstream clusters.
This will deploy 50 applications. The applications contain a medium amount of normal sized resources, e.g. 10 each.

• how long does it take for the git commit sha to show up in each GitRepo resource (duration seconds per repo)
• how long till bundle is created (per bundle)
• how long till bundledeployment is created (per bundledeployment)
• how long till agent finished install (per helm release)
• how long till gitrepo/cluster/clustergroup are updated
• number of reconciles (from prometheus)
• duration of reconciles (from prometheus)
• overall duration
• controllers memory usage before and after the test
• the amount of resources on the control plane before and after the test

Output the results in a useful format, e.g. JSON.

Future Work

• setup of test environment
• more test cases, e.g. one gitrepo, 50 gitrepos, 100 clusters
• measure the time it takes to update the Rancher UI
• measure CPU usage

[mmartin24]

FYI: just adding this tool I used in the past for performance measurement in case it can be found useful when it reaches the UI: https://grafana.com/products/cloud/k6/?src=k6io .

[shane-davidson]

Is there a difference between a single git repo with 10 fleet.yaml files or a single fleet.yaml file with 10x the resources inside it?

For the fleet deployment it would either result in a single bundle or multiple bundles.

Is there a performance difference? Is there a recommended way to deploy resources which is better for performance?

I guess the same question also applies to having multiple git repos -> single fleet.yaml file vs single git repo -> multiple fleet.yaml files

[manno]

#2133 (comment) is about writing a document about scaling.

@shane-davidson Yes, it makes a difference. For example multiple git repos mean more work for the gitops controller. A single large deployment deploys slowly, since nothing can be done in parallel.
I'd recommend less gitrepos and more bundles, but without a benchmark suite it's difficult to say. It also depends on your definition of performance.

Our talk on GitOps at Scale might be interesting:
https://youtu.be/1nhYdSy-VTE?si=Ao-hKHb-MLPhhVI9