Merge pull request #12880 from kalexand-rh/4.0_arch

4.0 arch draft

Author: kalexand-rh

_topic_map.yml

@@ -57,6 +57,13 @@ Topics:
 - Name: OpenShift CCS modular docs conventions
   File: mod-docs-conventions-ocp
 ---
+Name: Architecture
+Dir: architecture
+Distros: openshift-*
+Topics:
+- Name: Architecture
+  File: architecture
+---  
 Name: Authentication
 Dir: authentication
 Distros: openshift-*

architecture/architecture.adoc

@@ -0,0 +1,44 @@
+// This assembly is included in the following assemblies:
+//
+// * n/a
+
+[id='architecture']
+= {product-title} architecture
+include::modules/common-attributes.adoc[]
+:context: architecture
+toc::[]
+
+
+[IMPORTANT]
+====
+This assembly is a temporary placeholder to port the valid information from
+the 3.11 collection and include specific changes for 4.0 as that information
+becomes available.
+====
+
+include::modules/architecture-overview.adoc[leveloffset=+1]
+
+include::modules/installation-options.adoc[leveloffset=+2]
+
+include::modules/update-service-overview.adoc[leveloffset=+2]
+
+include::modules/node-management.adoc[leveloffset=+2]
+
+include::modules/node-types.adoc[leveloffset=+2]
+
+include::modules/operators-overview.adoc[leveloffset=+2]
+
+include::modules/abstraction-layers.adoc[leveloffset=+2]
+
+include::modules/machine-api-overview.adoc[leveloffset=+1]
+
+[[observability-architecture]]
+== Observability
+
+[IMPORTANT]
+====
+This section of the assembly is a placeholder for the Observability section,
+which will explain how monitoring, alerting, grafana, logging, and telemetry fit together.
+====
+
+include::modules/telemetry-service-overview.adoc[leveloffset=+2]

architecture/images/architecture_overview.png

@@ -0,0 +1,23 @@
+// Module included in the following assemblies:
+//
+// * architecture/architecture.adoc
+
+[id='abstraction-layers-{context}']
+= {product-title} abstraction layers
+
+The container service provides the abstraction for packaging and creating
+Linux-based, lightweight container images. Kubernetes provides the
+cluster management and orchestrates containers on multiple hosts.
+
+{product-title} adds:
+
+- Source code management, builds, and deployments for developers
+- Managing and promoting images at scale as they flow through your system
+- Application management at scale
+- Team and user tracking for organizing a large developer organization
+- Networking infrastructure that supports the cluster
+
+.{product-title} Architecture Overview
+image::../images/architecture_overview.png[{product-title} Architecture Overview]
+
+The cluster uses a combination of master and worker nodes.

modules/abstraction-layers.adoc

@@ -0,0 +1,80 @@
+// Module included in the following assemblies:
+//
+// * architecture/architecture.adoc
+
+[id='architecture-overview-{context}']
+= Architecture overview
+
+With {product-title} v4, the core story remains unchanged: {product-title} offers
+your developers a set of tools to evolve their applications under operational oversight
+and using Kubernetes to provide application infrastructure. The key change to v4 is
+that the infrastructure and its management are flexible, automated, and self-managing.
+
+A major difference between {product-title} v3 and v4 is that v4 uses Operators
+as both the fundamental unit of the product and an option for easily deploying
+and managing utilities that your apps use.
+
+{product-title} v4 runs on top of a Kubernetes cluster, with data about the
+objects stored in etcd, a reliable clustered key-value store. The cluster is 
+enhanced with standard components that you need to run your cluster, including 
+network, ingress, logging, and monitoring, that run as Operators to increase the
+ease and automation of installation, scaling, and maintenance.
+
+////
+The core services include:
+
+* Operators, which run the core {product-title} services.
+* REST APIs, which expose each of the core objects:
+** Containers and images, which are the building blocks for deploying your
+applications.
+** Pods and services, which containers use to communicate with each other and
+proxy connections.
+** Projects and users, which provide the space and means for communities to
+organize and manage their content together.
+** Builds and image streams allow you to
+build working images and react to new images.
+** Deployments, which expand support for the software development and deployment
+lifecycle.
+** Ingress and routes, which announce your service to the world.
+* Controllers, which read those REST APIs, apply changes to other objects, and
+report status or write back to the object.
+////
+
+{product-title} offers a catalog of supporting application infrastructure that
+includes:
+
+* Operators, which expose APIs that automate the complete component lifecycle 
+and include components like databases
+* Service bindings, which consume services that run outside the cluster
+* Templates, which are simple instant examples
+
+Users make calls to the REST API to change the state of the system. Controllers
+use the REST API to read the user's desired state and then try to bring the
+other parts of the system into sync. For example, when you request a build, the
+REST APIs create a `build` object. The build controller sees that a new build has been created, and
+runs a process on the cluster to perform that build. When the build completes,
+the controller updates the build object via the REST API and the user sees that
+their build is complete.
+
+The controller pattern means that much of the functionality in {product-title}
+is extensible. The way that builds are run and launched can be customized
+independently of how images are managed, or how deployments happen. The controllers
+perform the "business logic" of the system, taking user actions and
+transforming them into reality. By customizing those controllers or replacing
+them with your own logic, you can implement different behaviors. From a system
+administration perspective, this also means that you can use the API to script common
+administrative actions on a repeating schedule. Those scripts are also
+controllers that watch for changes and take action. {product-title} makes the
+ability to customize the cluster in this way a first-class behavior.
+
+To make this possible, controllers use a reliable stream of changes to the
+system to sync their view of the system with what users are doing. This event
+stream pushes changes from etcd to the REST API and then to the controllers as
+soon as changes occur so changes can efficiently ripple through the system.
+However, because failures can occur at any time, the controllers
+must also be able to get the latest state of the system at start up and confirm
+that everything is in the right state. This resynchronization is important
+because it means that even if something goes wrong, you can
+restart the affected components, and the system confirms its status before it
+continues. Because the controllers can always bring the system into sync, the
+system eventually converges to your intent.

modules/architecture-overview.adoc

@@ -4,12 +4,12 @@
 // * installation/installing-customizations-cloud.adoc
 
 [id='cloud-installations-{context}']
-= {product-title} clusters on clouds
+= {product-title} clusters on Installer Provisioned Infrastructure
 
 [IMPORTANT]
 ====
 In version {product-version}, you can install {product-title} on only Amazon
-Web Services (AWS) or your own hosts.
+Web Services (AWS).
 ====
 
 You can install either a standard cluster or a customized cluster. With a
@@ -19,8 +19,46 @@ cluster. These details include ...
 With a customized cluster, you can specify more details of the installation, 
 such as ...
 
+When you install {product-title} cluster with Installer Provisioned Infrastructure (IPI), you download the
+installer from link:try.openshift.com. This site manages:
+* REST API for accounts
+* Registry tokens, which are the pull secrets that you use to obtain the required
+components
+* Cluster registration, which associates the cluster identity to your Red Hat
+account to facilitate the gathering of usage metrics
+
+In {product-title} v4, the installer is a Go binary that performs a
+series of file transformations on a set of assets. When you use Installer 
+Provisioned Infrastructure, 
+you delegate the infrastructure bootstrapping and provisioning to the installer
+instead of doing it yourself. Because you do not use the installer to upgrade or
+update your cluster, if you do not highly customize your cluster, you run the
+installer only once.
+
+You use three sets of files during installation: an installation configuration
+file, Kubernetes manifests, and Ingition configurations for your machine types.
+
+The installation configuration file is transformed into Kubernetes manifests, and
+then the manifests are wrapped into Ignition configurations. The installer uses
+these Ingition configurations to create the cluster.
+
+The install configuration files are all pruned when you run the installer,
+so be sure to back up all configuration files that you want to use again.
+
 [IMPORTANT]
 ====
 You cannot modify the parameters that you set during installation. You have to
 ...
 ====
+
+////
+There are individual commands to perform the different actions in cluster creation
+if you want to try to make customizations, but you can run openshift-install
+create cluster to get the default cluster done quick.
+
+$ openshift-install --help
+$ openshift-install create install-config
+$ openshift-install create manifests
+$ openshift-install create ignition-configs
+$ openshift-install create cluster
+////

modules/cloud_installations.adoc

@@ -0,0 +1,40 @@
+// Module included in the following assemblies:
+//
+// * architecture/architecture.adoc
+
+[id='installation-options-{context}']
+= Installation options
+
+In {product-title} version 4.0, you can install only clusters that use
+Installer Provisioned Infrastructure in Amazon Web Services (AWS).
+These clusters use Red Hat CoreOS
+nodes as the operating system. Future versions of {product-title} will support
+clusters that use both Installer Provisioned Infrastructure
+and User Provisioned Infrastructure on more cloud providers and on bare metal.
+With all cluster types, you must use Red Hat CoreOS as the operating system for
+control plane nodes.
+////
+If you want to
+use any other cloud or install your cluster on-premise, use the bring your own
+infrastructure option to install your cluster on existing Red Hat Enterprise 
+Linux (RHEL) hosts.
+////
+
+Using Installer Provisioned Infrastructure offers full-stack automation to:
+
+* Manage compute
+* Manage operating system (RHCOS)
+* Manage control plane
+* Manage nodes
+
+////
+With the bring your own infrastructure option, you have more responsibilities.
+You must provide the hosts and update RHEL on them. {product-title} provides:
+
+* Managed control plane
+* Ansible to manage kubelet and container runtime
+////
+
+Installation and upgrade both use an Operator
+that constantly reconciles component versions as if it were any other Kubernetes
+controller.

modules/installation-options.adoc

@@ -0,0 +1,59 @@
+// Module included in the following assemblies:
+//
+// * architecture/architecture.adoc
+
+[id='machine-api-overview-{context}']
+= Machine API overview
+
+For {product-title} v4 clusters, the Machine API performs all node
+management actions after the cluster installation finishes. Because of this
+system, {product-title} v4 offers an elastic, dynamic provisioning
+method on top of public or private cloud infrastructure.
+
+The Machine API is a combination of primary resources that are based on the upstream
+link:https://github.com/kubernetes-sigs/cluster-api[Cluster API] project and
+custom {product-title} resources.
+
+The three primary resources are:
+
+`Machines`:: A fundamental unit that describes a `Node`. A `machine` has a
+class, which describes the types of compute nodes that are offered for different
+cloud platforms. For example, a `machine` type for a worker node on Amazon Web
+Services (AWS) might define a specific machine type and required metadata.
+`MachineClasses`:: A unit that defines a class of `machines` and facilitates
+configuration reuse across `machines` of the same class. This unit functions
+like a `StorageClass` for PersistentVolumeClaims.
+`MachineSets`:: Groups of machines. `MachineSets` are to `machines` as
+`ReplicaSets` are to `Pods`. If you need more `machines` or need to scale them down,
+you change the *replicas* field on the `MachineSet` to meet your compute need.
+
+
+The following custom resources add more capabilities to your cluster:
+
+`MachineAutoscaler`:: This resource automatically scales `machines` in
+a cloud. You can set the minimum and maximum scaling boundaries for nodes in a
+specified `MachineSet`, and the `MachineAutoscaler` maintains that range of nodes.
+The `MachineAutoscaler` object takes effect after a `ClusterAutoscaler` object
+exists. Both `ClusterAutoscaler` and `MachineAutoscaler` resources are made
+available by the `ClusterAutoscalerOperator`.
+`MachineHealthChecker`:: This resource detects when a machine is unhealthy,
+deletes it, and, on supported platforms, makes a new machine.
+`ClusterAutoscaler`:: This resource is based on the upstream
+link:https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler[Cluster Autoscaler]
+project. In the {product-title} implementation, it is integrated with the
+Cluster API by extending the `MachineSet` API.
+`ClusterAutoscalerOperator`:: Instead of interacting with the `ClusterAutoscaler`
+itself, you use its Operator. The `ClusterAutoscalerOperator` manages
+the `ClusterAutoscaler` deployment. With this Operator, you can set cluster-wide
+scaling limits for resources such as cores, nodes, memory, and GPU.
+and so on. You can set the priority so that the cluster prioritizes pods so that
+new nodes are not brought online for less important pods. You can also set the
+ScalingPolicy so you can scale up nodes but not scale them down.
+
+
+In {product-title} version 3.11, you could not roll out a multi-zone architecture easily because the cluster
+did not manage machine provisioning. It is easier in 4.0. Each `MachineSet` is scoped
+to a single zone, so the installer sends out `MachineSets` across availability zones
+on your behalf. And then because your compute is dynamic, and in 
+the face of a zone failure, you always have a zone for when you need to rebalance
+your machines. The autoscaler provides best-effort balancing over the life of a cluster.

modules/machine-api-overview.adoc

@@ -0,0 +1,48 @@
+// Module included in the following assemblies:
+//
+// * architecture/architecture.adoc
+
+[id='node-management-{context}']
+= Node management in {product-title}
+
+{product-title} version 4.0 integrates management of
+the container operating system and cluster management. Because the cluster manages
+its updates, including updates to Red Hat CoreOS on cluster nodes, {product-title} provides an opinionated
+lifecycle management experience that simplifies the orchestration of upgrades.
+
+{product-title} employs three DaemonSets and controllers to simplify node management:
+
+* The `machine-config-controller` coordinates machine upgrades.
+* The `machine-config-daemon` DaemonSet is a subset of the Ingition configuration that
+applies the specified machine configuration and controls kubelet configuration.
+* The `machine-config-server` DaemonSet provides the Ignition config to new hosts.
+
+These tools orchestrate operating system updates and configuration changes to
+the hosts by using standard Kuberentes-style constructs. A `machine-config-daemon`
+DaemonSet runs on each machine in the cluster and watches for changes in
+the machine configuration for it to apply. The machine configuration is a subset
+of the Ignition configuration. The `machine-config-daemon` reads the machine configuration to see
+if it needs to do an OSTree update, if it should apply a series of systemd
+kubelet file changes, configuration changes, or other changes to the
+operating system or {product-title} configuration.
+
+The masters also run the `machine-config-controller` process that monitors all of the cluster nodes
+and orchestrates their configuration updates. So if you try to apply
+an update or configuration change to a node on the cluster, the `machine-config-controller`
+directs a node to update. The node sees that it needs to change, drains off its
+pods, applies the update, and reboots. This process is key to the success of
+managing {product-title} and RHCOS updates together.
+
+The `machine-config-server` provides configurations to nodes as they join the
+cluster. It orchestrates configuration to nodes and changes to the operating system
+and is used in both cluster installation and node maintenance. The
+`machine-config-server` components upgrade the operating system and controls the Ignition
+configuration for nodes.
+
+////
+The `bootkube` process calls the `machine-config-server` component when the 
+{product-title} installer bootstraps the initial master node. After installation,
+the `machine-config-server` runs in the cluster.  It reads the `machine-config`
+custom resource definitions (CRDs) and serves the required Ignition configurations
+to new nodes when they join the cluster.
+////

modules/node-management.adoc

@@ -0,0 +1,23 @@
+// Module included in the following assemblies:
+//
+// * architecture/architecture.adoc
+
+[id='node-roles-{context}']
+= Node roles in {product-title}
+
+{product-title} assigns hosts different roles. These roles define the function
+of the node within the cluster. The cluster contains standard definitions for
+standard role types, such as bootstrap, master, and worker. 
+
+A node with the bootstrap role
+provides the initial configuration to clusters and is used only during initial
+configuration.
+
+Nodes with the master role run the cluster
+infrastructure and required components. Instead of being grouped into a `MachineSet`,
+they are a series of standalone machine API resources. Extra controls apply to
+master nodes to prevent you from deleting all master nodes and breaking your
+cluster.
+
+Nodes with the worker role drive compute workloads. Each type of worker node is
+governed by a specific machine pool that autoscales them.