Skip to content

Commit bd0d68a

Browse files
ahardin-rhahardin-rh
authored
Merge pull request #12900 from ahardin-rh/node-tuning-operator
Added node tuning operator content
2 parents 8161e14 + d24c85f commit bd0d68a

8 files changed

+313
-30
lines changed

‎_topic_map.yml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -248,8 +248,8 @@ Dir: scalability_and_performance
248248
Distros: openshift-origin, openshift-enterprise
249249
Topics:
250250
- Name: Recommended host practices
251-
File: recommended-host-practices
252-
- Name: Using the node tuning Operator
251+
File: recommended-host-practices
252+
- Name: Using the Node Tuning Operator
253253
File: using-node-tuning-operator
254254
- Name: Using Cluster Loader
255255
File: using-cluster-loader
@@ -384,4 +384,3 @@ Topics:
384384
File: efk-logging-exported-fields
385385
- Name: Uninstalling the EFK stack
386386
File: efk-logging-uninstall
387-

‎modules/accessing-an-example-cluster-node-tuning-operator-specification.adoc

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/using-node-tuning-operator.adoc
4+
5+
[id='accessing-an-example-node-tuning-operator-specification-{context}']
6+
= Accessing an example Node Tuning Operator specification
7+
8+
Use this process to access an example Node Tuning Operator specification.
9+
10+
.Procedure
11+
12+
. Run:
13+
+
14+
----
15+
$ oc get Tuned/default -o yaml
16+
----

‎modules/cluster-node-tuning-operator-default-profiles-set.adoc

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/using-node-tuning-operator.adoc
4+
5+
[id='custom-tuning-default-profiles-set-{context}']
6+
= Default profiles set on a cluster
7+
8+
The following are the default profiles set on a cluster.
9+
10+
----
11+
apiVersion: tuned.openshift.io/v1alpha1
12+
kind: Tuned
13+
metadata:
14+
name: default
15+
namespace: openshift-cluster-node-tuning-operator
16+
spec:
17+
profile:
18+
- name: "openshift"
19+
data: |
20+
[main]
21+
summary=Optimize systems running OpenShift (parent profile)
22+
include=${f:virt_check:virtual-guest:throughput-performance}
23+
[selinux]
24+
avc_cache_threshold=8192
25+
[net]
26+
nf_conntrack_hashsize=131072
27+
[sysctl]
28+
net.ipv4.ip_forward=1
29+
kernel.pid_max=>131072
30+
net.netfilter.nf_conntrack_max=1048576
31+
net.ipv4.neigh.default.gc_thresh1=8192
32+
net.ipv4.neigh.default.gc_thresh2=32768
33+
net.ipv4.neigh.default.gc_thresh3=65536
34+
net.ipv6.neigh.default.gc_thresh1=8192
35+
net.ipv6.neigh.default.gc_thresh2=32768
36+
net.ipv6.neigh.default.gc_thresh3=65536
37+
[sysfs]
38+
/sys/module/nvme_core/parameters/io_timeout=4294967295
39+
/sys/module/nvme_core/parameters/max_retries=10
40+
- name: "openshift-control-plane"
41+
data: |
42+
[main]
43+
summary=Optimize systems running OpenShift control plane
44+
include=openshift
45+
[sysctl]
46+
# ktune sysctl settings, maximizing i/o throughput
47+
#
48+
# Minimal preemption granularity for CPU-bound tasks:
49+
# (default: 1 msec# (1 + ilog(ncpus)), units: nanoseconds)
50+
kernel.sched_min_granularity_ns=10000000
51+
# The total time the scheduler will consider a migrated process
52+
# "cache hot" and thus less likely to be re-migrated
53+
# (system default is 500000, i.e. 0.5 ms)
54+
kernel.sched_migration_cost_ns=5000000
55+
# SCHED_OTHER wake-up granularity.
56+
#
57+
# Preemption granularity when tasks wake up. Lower the value to
58+
# improve wake-up latency and throughput for latency critical tasks.
59+
kernel.sched_wakeup_granularity_ns=4000000
60+
- name: "openshift-node"
61+
data: |
62+
[main]
63+
summary=Optimize systems running OpenShift nodes
64+
include=openshift
65+
[sysctl]
66+
net.ipv4.tcp_fastopen=3
67+
fs.inotify.max_user_watches=65536
68+
- name: "openshift-control-plane-es"
69+
data: |
70+
[main]
71+
summary=Optimize systems running ES on OpenShift control-plane
72+
include=openshift-control-plane
73+
[sysctl]
74+
vm.max_map_count=262144
75+
- name: "openshift-node-es"
76+
data: |
77+
[main]
78+
summary=Optimize systems running ES on OpenShift nodes
79+
include=openshift-node
80+
[sysctl]
81+
vm.max_map_count=262144
82+
recommend:
83+
- profile: "openshift-control-plane-es"
84+
priority: 10
85+
match:
86+
- label: "tuned.openshift.io/elasticsearch"
87+
type: "pod"
88+
match:
89+
- label: "node-role.kubernetes.io/master"
90+
- label: "node-role.kubernetes.io/infra"
91+
92+
- profile: "openshift-node-es"
93+
priority: 20
94+
match:
95+
- label: "tuned.openshift.io/elasticsearch"
96+
type: "pod"
97+
98+
- profile: "openshift-control-plane"
99+
priority: 30
100+
match:
101+
- label: "node-role.kubernetes.io/master"
102+
- label: "node-role.kubernetes.io/infra"
103+
104+
- profile: "openshift-node"
105+
priority: 40
106+
----

‎modules/custom-tuning-specification.adoc

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/using-node-tuning-operator.adoc
4+
5+
[id='custom-tuning-specification-{context}']
6+
= Custom tuning specification
7+
8+
[IMPORTANT]
9+
====
10+
Custom profiles are currently a Technology Preview feature.
11+
ifdef::openshift-enterprise[]
12+
Technology Preview features are not supported with Red Hat production service
13+
level agreements (SLAs), might not be functionally complete, and Red Hat does
14+
not recommend to use them for production. These features provide early access to
15+
upcoming product features, enabling customers to test functionality and provide
16+
feedback during the development process.
17+
18+
See the link:https://access.redhat.com/support/offerings/techpreview/[Red Hat
19+
Technology Preview features support scope] for more information.
20+
endif::[]
21+
====
22+
23+
The custom resource (CR) for the operator has two major sections. The first
24+
section, `profile:`, is a list of tuned profiles and their names. The second,
25+
`recommend:`, defines the profile selection logic.
26+
27+
Multiple custom tuning specifications can co-exist as multiple CRs in the
28+
operator's namespace. The existence of new CRs or the deletion of old CRs is
29+
detected by the Operator. All existing custom tuning specifications are merged
30+
and appropriate objects for the containerized tuned daemons are updated.
31+
32+
*profile: data*
33+
34+
The `profile:` section lists tuned profiles and their names.
35+
36+
----
37+
profile:
38+
- name: tuned_profile_1
39+
data: |
40+
# Tuned profile specification
41+
[main]
42+
summary=Description of tuned_profile_1 profile
43+
44+
[sysctl]
45+
net.ipv4.ip_forward=1
46+
# ... other sysctl's or other tuned daemon plugins supported by the containerized tuned
47+
48+
# ...
49+
50+
- name: tuned_profile_n
51+
data: |
52+
# Tuned profile specification
53+
[main]
54+
summary=Description of tuned_profile_n profile
55+
56+
# tuned_profile_n profile settings
57+
----
58+
59+
*Recommended profiles*
60+
61+
The `profile:` selection logic is defined by the `recommend` section of the CR:
62+
63+
----
64+
recommend:
65+
- match: # optional; if omitted, profile match is assumed unless a profile with a higher matches first
66+
<match> # an optional array
67+
priority: <priority> # profile ordering priority, lower numbers mean higher priority (0 is the highest priority)
68+
profile: <tuned_profile_name> # e.g. tuned_profile_1
69+
70+
# ...
71+
72+
- match:
73+
<match>
74+
priority: <priority>
75+
profile: <tuned_profile_name> # e.g. tuned_profile_n
76+
----
77+
78+
If `<match>` is omitted, a profile match (for example, `true`) is assumed.
79+
80+
`<match>` is an optional array recursively defined as follows:
81+
82+
----
83+
- label: <label_name> # node or pod label name
84+
value: <label_value> # optional node or pod label value; if omitted, the presence of <label_name> is enough to match
85+
type: <label_type> # optional node or pod type ("node" or "pod"); if omitted, "node" is assumed
86+
<match> # an optional <match> array
87+
----
88+
89+
If `<match>` is not omitted, all nested `<match>` sections must also evaluate to
90+
`true`. Otherwise, `false` is assumed and the profile with the respective
91+
`<match>` section will not be applied or recommended. Therefore, the nesting
92+
(child `<match>` sections) works as logical AND operator. Conversely, if any
93+
item of the `<match>` array matches, the entire `<match>` array evaluates to
94+
`true`. Therefore, the array acts as logical OR operator.
95+
96+
.Example
97+
98+
----
99+
- match:
100+
- label: tuned.openshift.io/elasticsearch
101+
match:
102+
- label: node-role.kubernetes.io/master
103+
- label: node-role.kubernetes.io/infra
104+
type: pod
105+
priority: 10
106+
profile: openshift-control-plane-es
107+
- match:
108+
- label: node-role.kubernetes.io/master
109+
- label: node-role.kubernetes.io/infra
110+
priority: 20
111+
profile: openshift-control-plane
112+
- priority: 30
113+
profile: openshift-node
114+
----
115+
116+
The CR above is translated for the containerized tuned daemon into its
117+
`recommend.conf` file based on the profile priorities. The profile with the
118+
highest priority (`10`) is `openshift-control-plane-es` and, therefore, it is
119+
considered first. The containerized tuned daemon running on a given node looks
120+
to see if there is a pod running on the same node with the
121+
`tuned.openshift.io/elasticsearch` label set. If not, the entire `<match>`
122+
section evaluates as `false`. If there is such a pod with the label, in order for
123+
the `<match>` section to evaluate to `true`, the node label also needs to be
124+
`node-role.kubernetes.io/master` or `node-role.kubernetes.io/infra`.
125+
126+
If the labels for the profile with priority `10` matched,
127+
`openshift-control-plane-es` profile is applied and no other profile is
128+
considered. If the node/pod label combination did not match, the second highest
129+
priority profile (`openshift-control-plane`) is considered. This profile is
130+
applied if the containerized tuned pod runs on a node with labels
131+
`node-role.kubernetes.io/master` or `node-role.kubernetes.io/infra`.
132+
133+
Finally, the profile `openshift-node` has the lowest priority of `30`. It lacks
134+
the `<match>` section and, therefore, will always match. It acts as a profile
135+
catch-all to set `openshift-node` profile, if no other profile with higher
136+
priority matches on a given node.

‎modules/deploying-node-tuning-operator.adoc

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎modules/node-tuning-operator-supported-tuned-daemon-plug-ins.adoc

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/using-node-tuning-operator.adoc
4+
5+
[id='supported-tuned-daemon-plug-ins-{context}']
6+
= Supported Tuned daemon plug-ins
7+
8+
Excluding the `[main]` section, the following Tuned plug-ins are supported when
9+
using custom profiles defined in the `profile:` section of the Tuned CR:
10+
11+
* audio
12+
* cpu
13+
* disk
14+
* eeepc_she
15+
* modules
16+
* mounts
17+
* net
18+
* scheduler
19+
* scsi_host
20+
* selinux
21+
* sysctl
22+
* sysfs
23+
* usb
24+
* video
25+
* vm
26+
27+
There is some dynamic tuning functionality provided by some of these plug-ins
28+
that is not supported. The following Tuned plug-ins are currently not supported:
29+
30+
* bootloader
31+
* script
32+
* systemd

‎modules/node-tuning-operator.adoc

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,17 @@
33
// * scalability_and_performance/using-node-tuning-operator.adoc
44

55
[id='about-node-tuning-operator-{context}']
6-
= Understanding the node tuning operator
6+
= About the Node Tuning Operator
77

8-
The node tuning operator helps you manage node-level tuning by orchestrating the
9-
tuned daemon.
8+
The Node Tuning Operator helps you manage node-level tuning by orchestrating the
9+
tuned daemon.The majority of high-performance applications require some level of
10+
kernel tuning. The Node Tuning Operator provides a unified management interface
11+
to users of node-level sysctls and more flexibility to add custom tuning, which
12+
is currently a Technology Preview feature, specified by user needs.The Operator
13+
manages the containerized tuned daemon for {product-title} as a Kubernetes
14+
DaemonSet. It ensures the custom tuning specification is passed to all
15+
containerized tuned daemons running in the cluster in the format that the
16+
daemons understand. The daemons run on all nodes in the cluster, one per node.
1017

11-
Here are more details about why it is used...
18+
The Node Tuning Operator is part of a standard {product-title} installation in
19+
version 4.0 and later.

‎scalability_and_performance/using-node-tuning-operator.adoc

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,20 @@
11
[id='using-node-tuning-operator']
2-
= Using the node tuning operator
2+
= Using the Node Tuning Operator
33
include::modules/common-attributes.adoc[]
44
:context: node-tuning-operator
55

66
toc::[]
77

88
{nbsp} +
9-
Learn about the node tuning operator and how you can use it to manage node-level
9+
Learn about the Node Tuning Operator and how you can use it to manage node-level
1010
tuning by orchestrating the tuned daemon.
1111

1212
include::modules/node-tuning-operator.adoc[leveloffset=+1]
1313

14-
include::modules/deploying-node-tuning-operator.adoc[leveloffset=+1]
14+
include::modules/accessing-an-example-cluster-node-tuning-operator-specification.adoc[leveloffset=+1]
15+
16+
include::modules/custom-tuning-specification.adoc[leveloffset=+1]
17+
18+
include::modules/cluster-node-tuning-operator-default-profiles-set.adoc[leveloffset=+1]
19+
20+
include::modules/node-tuning-operator-supported-tuned-daemon-plug-ins.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)