new assemblies and modules for cluster resoures

Author: mburke5678

_topic_map.yml

@@ -311,6 +311,12 @@ Topics:
   File: nodes-containers-port-forwarding
 - Name: Viewing system event information in a cluster
   File: nodes-containers-events
+- Name: Analyzing cluster resource levels
+  File: nodes-cluster-resource-levels
+- Name: Configuring cluster memory to meet container memory and risk requirements
+  File: nodes-cluster-resource-configure
+- Name: Configuring your cluster to place pods on overcommited nodes
+  File: nodes-cluster-overcommit
 ---
 Name: Logging
 Dir: logging

modules/nodes-cluster-overcommit-about.adoc

@@ -0,0 +1,19 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-about_{context}']
+= Understanding overcommitment in {product-title}
+
+Requests and limits enable administrators to allow and manage the overcommitment of resources on a node. The scheduler uses requests for scheduling your container and providing a minimum service guarantee. Limits constrain the amount of compute resource that may be consumed on your node.
+
+{product-title} administrators can control the level of overcommit and manage container density on nodes by configuring masters to override the ratio between request and limit set on developer containers. In conjunction with a per-project LimitRange specifying limits and defaults, this adjusts the container limit and request to achieve the desired level of overcommit.
+	
+[NOTE]
+====
+That these overrides have no effect if no limits have been set on containers. Create a LimitRange object with default limits (per individual project, or in the project template) in order to ensure that the overrides apply.
+====
+
+After these overrides, the container limits and requests must still be validated by any LimitRange objects in the project. It is possible, for example, for developers to specify a limit close to the minimum limit, and have the request then be overridden below the minimum limit, causing the pod to be forbidden. This unfortunate user experience should be addressed with future work, but for now, configure this capability and LimitRanges with caution.
+
+

modules/nodes-cluster-overcommit-configure-masters.adoc

@@ -0,0 +1,52 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-configure-masters_{context}']
+= Configuring masters for overcommitment
+
+{product-title} administrators control overcommit by configuring masters
+to override the ratio between request and limit set on developer
+containers. 
+
+.Prerequisites
+
+Because these overrides have no effect if no limits have
+been set on containers, you must create a *LimitRange*
+object with default limits, per individual project or in the
+project template, in order to ensure that the overrides apply.
+
+
+.Procedure
+
+To configure a master for overcommit:
+
+. Configuring the `*ClusterResourceOverride*` admission controller in the
+*_master-config.yaml_* as in the following example (reuse the existing configuration tree
+if it exists, or introduce absent elements as needed):
++
+[source,yaml]
+----
+  admissionConfig:
+    pluginConfig:
+      ClusterResourceOverride: <1>
+        configuration:
+          apiVersion: v1
+          kind: ClusterResourceOverrideConfig
+          memoryRequestToLimitPercent: 25 <2>
+          cpuRequestToLimitPercent: 25 <3>
+          limitCPUToMemoryPercent: 200 <4>
+----
+<1> This is the plug-in name; case matters and anything but an exact match for a plug-in name is ignored.
+<2> (optional, 1-100) If a container memory limit has been specified or defaulted, the memory request is overridden to this percentage of the limit.
+<3> (optional, 1-100) If a container CPU limit has been specified or defaulted, the CPU request is overridden to this percentage of the limit.
+<4> (optional, positive integer) If a container memory limit has been specified or defaulted, the CPU limit is overridden to a percentage of the memory limit, with a 100 percentage scaling 1Gi of RAM to equal 1 CPU core. This is processed prior to overriding CPU request (if configured).
+
+. Restart the master service:
++
+[source,bash]
+----
+# master-restart api
+# master-restart controllers
+----
+

modules/nodes-cluster-overcommit-configure-nodes.adoc

@@ -0,0 +1,43 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-configure-nodes_{context}']
+
+= Configuring nodes for overcommitment
+
+In an overcommitted environment, it is important to properly configure your node to provide best system behavior.
+
+When the node starts, it ensures that the kernel tunable flags for memory
+management are set properly. The kernel should never fail memory allocations
+unless it runs out of physical memory.
+
+To ensure this behavior, the node instructs the kernel to always overcommit
+memory:
+
+[source,bash]
+----
+$ sysctl -w vm.overcommit_memory=1
+----
+
+The node also instructs the kernel not to panic when it runs out of memory.
+Instead, the kernel OOM killer should kill processes based on priority:
+
+[source,bash]
+----
+$ sysctl -w vm.panic_on_oom=0
+----
+
+[NOTE]
+====
+The above flags should already be set on nodes, and no further action is
+required.
+====
+
+You can also perform the following configurations for each node:
+
+* Disable or enforce CPU limits using CPU CFS quotas
+
+* Reserve resources for system processes
+
+* Reserve memory across quality of service tiers

modules/nodes-cluster-overcommit-master-disable.adoc

@@ -0,0 +1,30 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-master-disable_{context}']
+= Disabling overcommitment for a project
+
+When configured, overcommitment can be disabled per-project.
+For example, you can allow infrastructure components to be configured independently of overcommitment.
+
+.Procedure
+
+To disable overcommitment in a project:
+
+. Edit the project object file
+
+. Add the following annotation:
++
+[source,bash]
+----
+quota.openshift.io/cluster-resource-override-enabled: "false"
+----
+
+. Create the project object:
++
+[source,bash]
+----
+oc create -f <file-name>.yaml
+----
+

modules/nodes-cluster-overcommit-master-disabling-swap.adoc

@@ -0,0 +1,25 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-disabling-swap_{context}']
+= Understanding swap memory and QOS
+
+You can disable swap by across all nodes in your cluster with a single command.
+
+.Procedure
+
+* Run the following command to disable swap:
++
+[source,bash]
+----
+$ swapoff -a
+----
+
+* Run the following command to enable swap:
++
+[source,bash]
+----
+$ swapon -a
+----
+

modules/nodes-cluster-overcommit-node-enforcing.adoc

@@ -0,0 +1,36 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-node-enforcing_{context}']
+
+= Disabling or enforcing CPU limits using CPU CFS quotas
+
+Nodes by default enforce specified CPU limits using the Completely Fair Scheduler (CFS) quota support in
+the Linux kernel. 
+
+.Procedure
+
+If you do not want to enforce CPU limits on the node, you can
+disable its enforcement by modifying the appropriate node configuration map 
+to include the following parameters:
+
+[source,yaml]
+----
+kubeletArguments:
+  cpu-cfs-quota:
+    - "false"
+----
+
+If CPU limit enforcement is disabled, it is important to understand the impact that will have on your node:
+
+- If a container makes a request for CPU, it will continue to be enforced by CFS
+shares in the Linux kernel.
+- If a container makes no explicit request for CPU, but it does specify a limit,
+the request will default to the specified limit, and be enforced by CFS shares
+in the Linux kernel.
+- If a container specifies both a request and a limit for CPU, the request will
+be enforced by CFS shares in the Linux kernel, and the limit will have no
+impact on the node.
+
+

modules/nodes-cluster-overcommit-node-memory.adoc

@@ -0,0 +1,43 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-node-memory_{context}']
+
+= Reserving memory across quality of service tiers
+
+You can use the `qos-reserved` parameter to specify a percentage of memory to be reserved
+by a pod in a particular QoS level. This feature attempts to reserve requested resources to exclude pods
+from lower OoS classes from using resources requested by pods in higher QoS classes.
+
+By reserving resources for higher QOS levels, pods that don't have resource limits are prevented from encroaching on the resources
+requested by pods at higher QoS levels.
+
+.Procedure
+
+To configure the `qos-reserved` parameter, edit the appropriate node configuration map.
+
+----
+kubeletArguments:
+  cgroups-per-qos:
+  - true
+  cgroup-driver:
+  - 'systemd'
+  cgroup-root:
+  - '/'
+  qos-reserved: <1>
+  - 'memory=50%'
+----
+<1> Specifies how pod resource requests are reserved at the QoS level.
+{product-title} uses the `qos-reserved` parameter as follows:
+- A value of `qos-reserved=memory=100%` will prevent the `Burstable` and `BestEffort` QOS classes from consuming memory
+that was requested by a higher QoS class. This increases the risk of inducing OOM
+on `BestEffort` and `Burstable` workloads in favor of increasing memory resource guarantees
+for `Guaranteed` and `Burstable` workloads.
+- A value of `qos-reserved=memory=50%` will allow the `Burstable` and `BestEffort` QOS classes
+to consume half of the memory requested by a higher QoS class.
+- A value of `qos-reserved=memory=0%`
+will allow a `Burstable` and `BestEffort` QoS classes to consume up to the full node
+allocatable amount if available, but increases the risk that a `Guaranteed` workload
+will not have access to requested memory. This condition effectively disables this feature.
+

modules/nodes-cluster-overcommit-node-resources.adoc

@@ -0,0 +1,17 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-node-resources_{context}']
+
+= Reserving resources for system processes
+
+To provide more reliable scheduling and minimize node resource overcommitment, 
+each node can reserve a portion of its resources for use by system daemons 
+that are required to run on your node for your cluster to function (*sshd*, *docker*, etc.). 
+In particular, it is recommended that you reserve resources for incompressible resources such as memory.
+
+.Procedure
+
+To explicitly reserve resources for non-pod processes, allocate node resources by specifying resources
+available for scheduling. See Allocating Node Resources for more details.

modules/nodes-cluster-overcommit-qos-about.adoc

@@ -0,0 +1,101 @@
+// Module included in the following assemblies:
+//
+// * nodes/nodes-cluster-overcommit.adoc
+
+[id='nodes-cluster-overcommit-qos-about_{context}']
+= Understanding overcomitment and quality of service classes
+
+A node is _overcommitted_ when it has a pod scheduled that makes no request, or
+when the sum of limits across all pods on that node exceeds available machine
+capacity.
+
+In an overcommitted environment, it is possible that the pods on the node will
+attempt to use more compute resource than is available at any given point in
+time. When this occurs, the node must give priority to one pod over another. The
+facility used to make this decision is referred to as a Quality of Service (QoS)
+Class.
+
+For each compute resource, a container is divided into one of three QoS classes
+with decreasing order of priority:
+
+.Quality of Service Classes
+[options="header",cols="1,1,5"]
+|===
+|Priority |Class Name |Description
+
+|1 (highest)
+|*Guaranteed*
+|If limits and optionally requests are set (not equal to 0) for all resources
+and they are equal, then the container is classified as *Guaranteed*.
+
+|2
+|*Burstable*
+|If requests and optionally limits are set (not equal to 0) for all resources,
+and they are not equal, then the container is classified as *Burstable*.
+
+|3 (lowest)
+|*BestEffort*
+|If requests and limits are not set for any of the resources, then the container
+is classified as *BestEffort*.
+|===
+
+Memory is an incompressible resource, so in low memory situations, containers
+that have the lowest priority are terminated first:
+
+- *Guaranteed* containers are considered top priority, and are guaranteed to
+only be terminated if they exceed their limits, or if the system is under memory
+pressure and there are no lower priority containers that can be evicted.
+- *Burstable* containers under system memory pressure are more likely to be
+terminated once they exceed their requests and no other *BestEffort* containers
+exist.
+- *BestEffort* containers are treated with the lowest priority. Processes in
+these containers are first to be terminated if the system runs out of memory.
+
+[[nodes-cluster-overcommit-qos-about-reserve]]
+== Understanding how to reserve memory across quality of service tiers
+
+You can use the `qos-reserved` parameter to specify a percentage of memory to be reserved
+by a pod in a particular QoS level. This feature attempts to reserve requested resources to exclude pods
+from lower OoS classes from using resources requested by pods in higher QoS classes.
+
+{product-title} uses the `qos-reserved` parameter as follows:
+
+- A value of `qos-reserved=memory=100%` will prevent the `Burstable` and `BestEffort` QOS classes from consuming memory
+that was requested by a higher QoS class. This increases the risk of inducing OOM
+on `BestEffort` and `Burstable` workloads in favor of increasing memory resource guarantees
+for `Guaranteed` and `Burstable` workloads.
+
+- A value of `qos-reserved=memory=50%` will allow the `Burstable` and `BestEffort` QOS classes
+to consume half of the memory requested by a higher QoS class.
+
+- A value of `qos-reserved=memory=0%`
+will allow a `Burstable` and `BestEffort` QoS classes to consume up to the full node
+allocatable amount if available, but increases the risk that a `Guaranteed` workload
+will not have access to requested memory. This condition effectively disables this feature.
+
+[[nodes-cluster-overcommit-qos-about-swap]]
+= Understanding swap memory and QOS
+
+You can disable swap by default on your nodes in order to preserve quality of
+service (QOS) guarantees. Otherwise, physical resources on a node can oversubscribe,
+affecting the resource guarantees the Kubernetes scheduler makes during pod
+placement.
+
+For example, if two guaranteed pods have reached their memory limit, each
+container could start using swap memory. Eventually, if there is not enough swap
+space, processes in the pods can be terminated due to the system being
+oversubscribed.
+
+Failing to disable swap results in nodes not recognizing that they are
+experiencing *MemoryPressure*, resulting in pods not receiving the memory they
+made in their scheduling request. As a result, additional pods are placed on the
+node to further increase memory pressure, ultimately increasing your risk of
+experiencing a system out of memory (OOM) event.
+
+[IMPORTANT]
+====
+If swap is enabled, any out-of-resource handling eviction thresholds for available memory will not work as
+expected. Take advantage of out-of-resource handling to allow pods to be evicted
+from a node when it is under memory pressure, and rescheduled on an alternative
+node that has no such pressure.
+====