[Umbrella] Performance optimizations planned for v1.15

[zach593]

Features that can be added before the v1.15 release deadline

1. Since we have disabled deepcopy during list operations in disable deepcopy for list action #5813 , we should enable the MutationDetector in our e2e tests to prevent any future code from mutating the cache.

2. Continue adding indices to improve list performance wherever applicable. [Umbrella] Use index to optimize list operations #6530

3. Introduce the controller-runtime priority queue. This enhancement allows controllers built on controller-runtime to prioritize the latest changes after a restart or leader transition, deferring the processing of inevitable create events. This can significantly reduce downtime during service restarts and failovers.

   • For this feature, we will likely need detailed test reports to ensure it does not introduce negative side effects.

Features that may progress before the v1.15 release deadline but are unlikely to be completed

1. Where feasible, migrate async-worker to use controller-runtime controllers. Replace asyncWorker with controller-runtime reconciler #6251

Features unlikely to start before the v1.15 release deadline but are likely to be pursued in the future

1. Add automated e2e benchmark tests in GitHub CI.
2. Introduce controller-runtime warmup. This feature is still under development ✨ Implement warm replica support for controllers kubernetes-sigs/controller-runtime#3192. Combined with the priority queue, it could theoretically reduce downtime to zero during multi-replica leader transitions, which is especially important in large-scale clusters.

---

The above list includes what I am currently aware of. If there are any omissions, please feel free to add or suggest them.

[zach593]

I will create separate issues for each task this week.

[RainbowMango]

Great thanks @zach593 for your hard work~ Take care!

[RainbowMango]

Speaking of the flags in karmada-controller-manager, like:

      --concurrent-cluster-propagation-policy-syncs int                                                                                                                                                                      
                The number of ClusterPropagationPolicy that are allowed to sync concurrently. (default 1)
      --concurrent-cluster-syncs int                                                                                                                                                                                         
                The number of Clusters that are allowed to sync concurrently. (default 5)
      --concurrent-clusterresourcebinding-syncs int                                                                                                                                                                          
                The number of ClusterResourceBindings that are allowed to sync concurrently. (default 5)
      --concurrent-dependent-resource-syncs int                                                                                                                                                                              
                The number of dependent resource that are allowed to sync concurrently. (default 2)
      --concurrent-namespace-syncs int                                                                                                                                                                                       
                The number of Namespaces that are allowed to sync concurrently. (default 1)
      --concurrent-propagation-policy-syncs int                                                                                                                                                                              
                The number of PropagationPolicy that are allowed to sync concurrently. (default 1)
      --concurrent-resource-template-syncs int                                                                                                                                                                               
                The number of resource templates that are allowed to sync concurrently. (default 5)
      --concurrent-resourcebinding-syncs int                                                                                                                                                                                 
                The number of ResourceBindings that are allowed to sync concurrently. (default 5)
      --concurrent-work-syncs int                                                                                                                                                                                            
                The number of Works that are allowed to sync concurrently. (default 5)

1. Shall we revisit the default values? I'm not sure if the default values of these parameters are overly conservative.
2. Shall we enrich the usage to add more instructions, such as when people need to consider enlarging the value?
3. Shall we group all these performance-related flags into a dedicated group like Performance flags?

[godwinpang]

fwiw kubernetes-sigs/controller-runtime#3192 has been merged