Skip to content

Commit 4252ee4

Browse files
xenolinuxxenolinux
authored
Merge pull request #87314 from kquinn1204/TELCODOCS-2149
TELCODOCS-2149 OCPBUGS-28647 tuned deferred updates
2 parents cfe106a + c044144 commit 4252ee4

File tree

3 files changed

+283
-0
lines changed

3 files changed

+283
-0
lines changed

‎modules/defer-application-tuning-proc.adoc

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/using-node-tuning-operator.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="defer-application-of-tuning-changes-example_{context}"]
7+
= Deferring application of tuning changes: An example
8+
9+
The following worked example describes how to defer the application of tuning changes by using the Node Tuning Operator.
10+
11+
.Prerequisites
12+
* You have `cluster-admin` role access.
13+
* You have applied a performance profile to your cluster.
14+
* A `MachineConfigPool` resource, for example, `worker-cnf` is configured to ensure that the profile is only applied to the designated nodes.
15+
16+
.Procedure
17+
18+
. Check what profiles are currently applied to your cluster by running the following command:
19+
+
20+
[source,shell]
21+
----
22+
$ oc -n openshift-cluster-node-tuning-operator get tuned
23+
----
24+
+
25+
.Example output
26+
[source,shell]
27+
----
28+
NAME AGE
29+
default 63m
30+
openshift-node-performance-performance 21m
31+
----
32+
33+
. Check the machine config pools in your cluster by running the following command:
34+
+
35+
[source,shell]
36+
----
37+
$ oc get mcp
38+
----
39+
+
40+
.Example output
41+
[source,shell]
42+
----
43+
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
44+
master rendered-master-79a26af9f78ced61fa8ccd309d3c859c True False False 3 3 3 0 157m
45+
worker rendered-worker-d9352e91a1b14de7ef453fa54480ce0e True False False 2 2 2 0 157m
46+
worker-cnf rendered-worker-cnf-f398fc4fcb2b20104a51e744b8247272 True False False 1 1 1 0 92m
47+
----
48+
49+
. Describe the current applied performance profile by running the following command:
50+
+
51+
[source,shell]
52+
----
53+
$ oc describe performanceprofile performance | grep Tuned
54+
----
55+
+
56+
.Example output
57+
[source,shell]
58+
----
59+
Tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-performance
60+
----
61+
62+
. Verify the existing value of the `kernel.shmmni` sysctl parameter:
63+
64+
.. Run the following command to display the node names:
65+
+
66+
[source,shell]
67+
----
68+
$ oc get nodes
69+
----
70+
+
71+
.Example output
72+
[source,shell]
73+
----
74+
NAME STATUS ROLES AGE VERSION
75+
ip-10-0-26-151.ec2.internal Ready worker,worker-cnf 116m v1.30.6
76+
ip-10-0-46-60.ec2.internal Ready worker 115m v1.30.6
77+
ip-10-0-52-141.ec2.internal Ready control-plane,master 123m v1.30.6
78+
ip-10-0-6-97.ec2.internal Ready control-plane,master 121m v1.30.6
79+
ip-10-0-86-145.ec2.internal Ready worker 117m v1.30.6
80+
ip-10-0-92-228.ec2.internal Ready control-plane,master 123m v1.30.6
81+
----
82+
83+
.. Run the following command to display the current value of the `kernel.shmmni` sysctl parameters on the node `ip-10-0-32-74.ec2.internal`:
84+
+
85+
[source,shell]
86+
----
87+
$ oc debug node/ip-10-0-26-151.ec2.internal -q -- chroot host sysctl kernel.shmmni
88+
----
89+
+
90+
.Example output
91+
[source,shell]
92+
----
93+
kernel.shmmni = 4096
94+
----
95+
96+
. Create a profile patch, for example, `perf-patch.yaml` that changes the `kernel.shmmni` sysctl parameter to `8192`. Defer the application of the change to a new manual restart by using the `always` method by applying the following configuration:
97+
+
98+
[source,yaml]
99+
----
100+
apiVersion: tuned.openshift.io/v1
101+
kind: Tuned
102+
metadata:
103+
name: performance-patch
104+
namespace: openshift-cluster-node-tuning-operator
105+
annotations:
106+
tuned.openshift.io/deferred: "always"
107+
spec:
108+
profile:
109+
- name: performance-patch
110+
data: |
111+
[main]
112+
summary=Configuration changes profile inherited from performance created tuned
113+
include=openshift-node-performance-performance <1>
114+
[sysctl]
115+
kernel.shmmni=8192 <2>
116+
recommend:
117+
- machineConfigLabels:
118+
machineconfiguration.openshift.io/role: worker-cnf <3>
119+
priority: 19
120+
profile: performance-patch
121+
----
122+
+
123+
<1> The `include` directive is used to inherit the `openshift-node-performance-performance` profile. This is a best practice to ensure that the profile is not missing any required settings.
124+
<2> The `kernel.shmmni` sysctl parameter is being changed to `8192`.
125+
<3> The `machineConfigLabels` field is used to target the `worker-cnf` role.
126+
127+
. Apply the profile patch by running the following command:
128+
+
129+
[source,shell]
130+
----
131+
$ oc apply -f perf-patch.yaml
132+
----
133+
134+
. Run the following command to verify that the profile patch is waiting for the next node restart:
135+
+
136+
[source,shell]
137+
----
138+
$ oc -n openshift-cluster-node-tuning-operator get profile
139+
----
140+
+
141+
.Example output
142+
[source,shell]
143+
----
144+
NAME TUNED APPLIED DEGRADED MESSAGE AGE
145+
ip-10-0-26-151.ec2.internal performance-patch False True The TuneD daemon profile is waiting for the next node restart: performance-patch 126m
146+
ip-10-0-46-60.ec2.internal openshift-node True False TuneD profile applied. 125m
147+
ip-10-0-52-141.ec2.internal openshift-control-plane True False TuneD profile applied. 130m
148+
ip-10-0-6-97.ec2.internal openshift-control-plane True False TuneD profile applied. 130m
149+
ip-10-0-86-145.ec2.internal openshift-node True False TuneD profile applied. 126m
150+
ip-10-0-92-228.ec2.internal openshift-control-plane True False TuneD profile applied. 130m
151+
----
152+
153+
. Confirm the value of the `kernel.shmmni` sysctl parameter remain unchanged before a restart:
154+
155+
.. Run the following command to confirm that the application of the `performance-patch` change to the `kernel.shmmni` sysctl parameter on the node `ip-10-0-26-151.ec2.internal` is not applied:
156+
+
157+
[source,shell]
158+
----
159+
$ oc debug node/ip-10-0-26-151.ec2.internal -q -- chroot host sysctl kernel.shmmni
160+
----
161+
+
162+
.Example output
163+
[source,shell]
164+
----
165+
kernel.shmmni = 4096
166+
----
167+
168+
. Restart the node `ip-10-0-26-151.ec2.internal` to apply the required changes by running the following command:
169+
+
170+
[source,shell]
171+
----
172+
$ oc debug node/ip-10-0-26-151.ec2.internal -q -- chroot host reboot&
173+
----
174+
175+
. In another terminal window, run the following command to verify that the node has restarted:
176+
+
177+
[source,shell]
178+
----
179+
$ watch oc get nodes
180+
----
181+
+
182+
Wait for the node `ip-10-0-26-151.ec2.internal` to transition back to the `Ready` state.
183+
184+
. Run the following command to verify that the profile patch is waiting for the next node restart:
185+
+
186+
[source,shell]
187+
----
188+
$ oc -n openshift-cluster-node-tuning-operator get profile
189+
----
190+
+
191+
.Example output
192+
[source,shell]
193+
----
194+
NAME TUNED APPLIED DEGRADED MESSAGE AGE
195+
ip-10-0-20-251.ec2.internal performance-patch True False TuneD profile applied. 3h3m
196+
ip-10-0-30-148.ec2.internal openshift-control-plane True False TuneD profile applied. 3h8m
197+
ip-10-0-32-74.ec2.internal openshift-node True True TuneD profile applied. 179m
198+
ip-10-0-33-49.ec2.internal openshift-control-plane True False TuneD profile applied. 3h8m
199+
ip-10-0-84-72.ec2.internal openshift-control-plane True False TuneD profile applied. 3h8m
200+
ip-10-0-93-89.ec2.internal openshift-node True False TuneD profile applied. 179m
201+
----
202+
203+
. Check that the value of the `kernel.shmmni` sysctl parameter have changed after the restart:
204+
205+
.. Run the following command to verify that the `kernel.shmmni` sysctl parameter change has been applied on the node `ip-10-0-32-74.ec2.internal`:
206+
+
207+
[source,shell]
208+
----
209+
$ oc debug node/ip-10-0-32-74.ec2.internal -q -- chroot host sysctl kernel.shmmni
210+
----
211+
+
212+
.Example output
213+
[source,shell]
214+
----
215+
kernel.shmmni = 8192
216+
----
217+
218+
[NOTE]
219+
====
220+
An additional restart results in the restoration of the original value of the `kernel.shmmni` sysctl parameter.
221+
====

‎modules/defer-applicaton-tuning-example.adoc

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/using-node-tuning-operator.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="defer-application-of-tuning-changes_{context}"]
7+
= Deferring application of tuning changes
8+
9+
As an administrator, use the Node Tuning Operator (NTO) to update custom resources (CRs) on a running system and make tuning changes. For example, they can update or add a sysctl parameter to the [sysctl] section of the tuned object. When administrators apply a tuning change, the NTO prompts TuneD to reprocess all configurations, causing the tuned process to roll back all tuning and then reapply it.
10+
11+
Latency-sensitive applications may not tolerate the removal and reapplication of the tuned profile, as it can briefly disrupt performance. This is particularly critical for configurations that partition CPUs and manage process or interrupt affinity using the performance profile. To avoid this issue, {product-title} introduced new methods for applying tuning changes. Before {product-title} 4.17, the only available method, immediate, applied changes instantly, often triggering a tuned restart.
12+
13+
The following additional methods are supported:
14+
15+
* `always`: Every change is applied at the next node restart.
16+
* `update`: When a tuning change modifies a tuned profile, it is applied immediately by default and takes effect as soon as possible. When a tuning change does not cause a tuned profile to change and its values are modified in place, it is treated as always.
17+
18+
Enable this feature by adding the annotation `tuned.openshift.io/deferred`. The following table summarizes the possible values for the annotation:
19+
20+
[cols="3,3",options="header"]
21+
|===
22+
|Annotation value | Description
23+
|missing | The change is applied immediately.
24+
|always | The change is applied at the next node restart.
25+
|update | The change is applied immediately if it causes a profile change, otherwise at the next node restart.
26+
|===
27+
28+
The following example demonstrates how to apply a change to the `kernel.shmmni` sysctl parameter by using the `always` method:
29+
30+
.Example
31+
[source,yaml]
32+
----
33+
apiVersion: tuned.openshift.io/v1
34+
kind: Tuned
35+
metadata:
36+
name: performance-patch
37+
namespace: openshift-cluster-node-tuning-operator
38+
annotations:
39+
tuned.openshift.io/deferred: "always"
40+
spec:
41+
profile:
42+
- name: performance-patch
43+
data: |
44+
[main]
45+
summary=Configuration changes profile inherited from performance created tuned
46+
include=openshift-node-performance-performance <1>
47+
[sysctl]
48+
kernel.shmmni=8192 <2>
49+
recommend:
50+
- machineConfigLabels:
51+
machineconfiguration.openshift.io/role: worker-cnf <3>
52+
priority: 19
53+
profile: performance-patch
54+
----
55+
56+
<1> The `include` directive is used to inherit the `openshift-node-performance-performance` profile. This is a best practice to ensure that the profile is not missing any required settings.
57+
<2> The `kernel.shmmni` sysctl parameter is being changed to `8192`.
58+
<3> The `machineConfigLabels` field is used to target the `worker-cnf` role. Configure a `MachineConfigPool` resource to ensure the profile is applied only to the correct nodes.

‎scalability_and_performance/using-node-tuning-operator.adoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ include::modules/custom-tuning-specification.adoc[leveloffset=+1]
2121

2222
include::modules/custom-tuning-example.adoc[leveloffset=+1]
2323

24+
include::modules/defer-applicaton-tuning-example.adoc[leveloffset=+1]
25+
26+
include::modules/defer-application-tuning-proc.adoc[leveloffset=+2]
27+
2428
include::modules/node-tuning-operator-supported-tuned-daemon-plug-ins.adoc[leveloffset=+1]
2529

2630
include::modules/node-tuning-hosted-cluster.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)