-
Notifications
You must be signed in to change notification settings - Fork 5
Description
How to categorize this issue?
/area auto-scaling|
/kind enhancement
What would you like to be added:
We need the ability to generate Scaling Scenario's from real-world cluster. For now, we propose suggesting only gardener clusters, but tool must be generic enough to support scaling scenario generation for any cluster manager (cluster-api, etc), not just gardener.
CLI Tool scadctl genscenario
NOTE: scadctl genscenario
is an auxiliary command for the scaling-advisor ctl client scadctl
that generates scaling advisor scenarios from gardener shoot clusters.
Code Placement
- Introduce cobra command
genscenario
insidetools/cmd/scenario/genscenario.go
- Register cobra command into root.
- gardener cluster manager scenario generation code should go into
tools/cmd/scenario/gardener
. scenario generation for other cluster managers in the future (cluster-api
) can go into subpackages like:client/cmd/scenario/<cluster-manager>
. Ex:client/cmd/scenario/clusterapi
Command Overview
scadctl genscenario <cluster-manager> <cluster-manager-specific-options>
- For gardener clusters:
scadctl genscenario gardener -l <landscape> -p <project> -t <shoot-name> -d <scenario-dir>
. # capture cluster state and generates scenarios from live cluster
- For gardener clusters:
scadctl genscenario -m gardener -l <landscape> -p <project> -t <shoot-name> -d <scenario-dir>
- generates viewer kubeconfig for shoot data planea and control plane into OS
tmp/full-qualified-clustername
directory if not already generated.- Delegate to gardenctl CLI tool for this or use HTTP request to generate kubeconfigs. Do not introduce gardener code dependency.
- Example of
full-qualified-clustername -> dev:i034796:aw
- Connect to cluster and create a
svcapi.ClusterSnapshot
. (See Scaling Advisor Service API) - Connect to control cluster, get the
Worker
object and then create thesacorev1alpha1.aClusterScalingConstraints
(See Scaling Advisor Common Core API) - (optional) It should obfuscate the node names and pod names and pool names. Use generated human names component.
- Create the
scenario-dir
if it does not exist. - Persist the the
sacorev1alpha1.ClusterScalingConstraints
intoscenario-dir/<full-qualified-cluster-name>/cluster-scaling-constraints.json
. - It should now create cluster snapshot variants:
scenario-dir/clusterName/snapshot-ddMMYY-latest-incremental.json
- From the captured cluster snapshot. remove the latest created node , unassign the pods from this node. Rewrite the cluster snapshot
scenario-dir/clusterName/snapshot-ddMMYY-latest-5.json
- From the captured cluster snapshot. remove the latest 5 created node , unassign the pods from this node. Rewrite the cluster snapshot
- (Redo the above for 5-10-20 nodes)
- The
scenario-dir
can be checked intogithub.tools.sap/kubernetes/gardener-scaling-tools/scenarios
to allow other gardener members to reply and reproduce issues in scenarios via thescalebench
tool described later. (If scenarios are too large, we can either compress and. check them in or we can attach them as github release artifacts instead of direct source-checkin)
/scenario-dir
/<full-qualified-cluster-name>
/cluster-scaling-constraints.json
/snapshot-ddMMYY-latest-incremental.json
/snapshot-ddMMYY-latest-5.json
...
Why is this needed:
We need to test scaling advisor (and other scalers) against actual scaling scenarios. For the gardener cluster-manager, we need the ability to adapt the gardener shoot worker object and the state of the shoot cluster into a svcapi.ScalingAdviceRequest
that the scaling-advisor can be invoked with.