Skip to content

Commit 38743da

Browse files
paigecalvertpaigecalvert
authored
Merge pull request #3216 from replicatedhq/122803
HOLD FOR RELEASE: use cli to get commands for node joins
2 parents 169cccf + 8a9e64b commit 38743da

File tree

3 files changed

+67
-62
lines changed

3 files changed

+67
-62
lines changed

‎docs/enterprise/embedded-manage-nodes.mdx

Lines changed: 67 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -8,88 +8,99 @@ This topic describes managing nodes in clusters created with Replicated Embedded
88

99
Multi-node clusters with Embedded Cluster have the following limitations:
1010

11-
* Support for multi-node clusters with Embedded Cluster is Beta. Only single-node embedded clusters are Generally Available (GA).
12-
13-
* High availability for Embedded Cluster in an Alpha feature. This feature is subject to change, including breaking changes. For more information about this feature, reach out to Alex Parker at [alexp@replicated.com](mailto:alexp@replicated.com).
11+
* Setting node roles with the Embedded Cluster Config [roles](/reference/embedded-config#roles) key is Beta.
1412

1513
* The same Embedded Cluster data directory used at installation is used for all nodes joined to the cluster. This is either the default `/var/lib/embedded-cluster` directory or the directory set with the [`--data-dir`](/reference/embedded-cluster-install#flags) flag. You cannot choose a different data directory for Embedded Cluster when joining nodes.
1614

1715
* More than one controller node should not be joined at the same time. When joining a controller node, a warning is printed that explains that the user should not attempt to join another node until the controller node joins successfully.
1816

19-
* Setting node roles with the Embedded Cluster Config [roles](/reference/embedded-config#roles) key is Beta.
17+
* The `join print-command` command always returns the commands for joining a node with the controller role. It does not support printing the join command for any custom node roles defined in the Embedded Cluster Config `roles` key. See [Automate Controller Node Joins](#automate-node-joins) below.
2018

2119
## Requirement
2220

2321
To deploy multi-node clusters with Embedded Cluster, the **Multi-node Cluster (Embedded Cluster only)** license field must be enabled for the customer. For more information about managing customer licenses, see [Create and Manage Customers](/vendor/releases-creating-customer).
2422

25-
## Add Nodes to a Cluster (Beta) {#add-nodes}
26-
27-
You can add nodes to create a multi-node cluster in online (internet-connected) and air-gapped (limited or no outbound internet access) environments. The Admin Console provides the join command that you use to join nodes to the cluster.
23+
## Add Nodes to a Cluster {#add-nodes}
2824

29-
:::note
30-
Multi-node clusters are not highly available by default. For information about enabling high availability, see [Enable High Availability for Multi-Node Clusters (Alpha)](#ha) below.
31-
:::
25+
This section describes how to add nodes to a cluster with Embedded Cluster.
3226

33-
To add nodes to a cluster:
27+
To add a node to a cluster with Embedded Cluster:
3428

3529
1. (Optional) In the Embedded Cluster Config, configure the `roles` key to customize node roles. For more information, see [roles](/reference/embedded-config#roles) in _Embedded Cluster Config_. When you are done, create and promote a new release with the updated Config.
3630

37-
1. Do one of the following to get the join command from the Admin Console:
31+
1. Do one of the following:
3832

39-
1. To add nodes during the application installation process, follow the steps in [Online Installation with Embedded Cluster](/enterprise/installing-embedded) or [Air Gap Installation with Embedded Cluster](/enterprise/installing-embedded-air-gap) to install. A **Nodes** screen is displayed as part of the installation flow in the Admin Console that allows you to choose a node role and copy the relevant join command.
33+
1. Follow the steps in [Online Installation with Embedded Cluster](/enterprise/installing-embedded) or [Air Gap Installation with Embedded Cluster](/enterprise/installing-embedded-air-gap)to install. When you see the **Nodes** screen as part of the installation flow in the Admin Console, continue to the next step.
4034

41-
1. Otherwise, if you have already installed the application:
35+
1. Otherwise, to add a node to an existing cluster:
4236

43-
1. Log in to the Admin Console.
44-
45-
1. If you promoted a new release that configures the `roles` key in the Embedded Cluster Config, update the instance to the new version. See [Perform Updates in Embedded Clusters](/enterprise/updating-embedded).
46-
47-
1. Go to **Cluster Management > Add node** at the top of the page.
37+
1. Log in to the Admin Console.
38+
39+
1. If you promoted a new release that configures the `roles` key in the Embedded Cluster Config, update the instance to the new version. See [Perform Updates in Embedded Clusters](/enterprise/updating-embedded).
40+
41+
1. Go to **Cluster Management > Add node** at the top of the page.
42+
43+
1. If you configured the `roles` key to customize node roles, select one or more roles for the node.
4844

49-
<img alt="Add node page in the Admin Console" src="/images/admin-console-add-node.png" width="600px"/>
45+
The Admin Console page is updated to display the commands for downloading the Embedded Cluster binary, extracting the binary, and joining the node to the cluster based on the roles you selected. Keep this Admin Console page open for the next steps.
5046

51-
[View a larger version of this image](/images/admin-console-add-node.png)
52-
53-
1. Either on the Admin Console **Nodes** screen that is displayed during installation or in the **Add a Node** dialog, select one or more roles for the new node that you will join. Copy the join command.
47+
:::note
48+
The role cannot be changed after a node is added. If you need to change a node’s role, reset the node and add it again with the new role.
49+
:::
50+
51+
1. SSH onto the node that you want to join.
5452

55-
Note the following:
53+
1. Copy the curl command provided in the Admin Console to download the Embedded Cluster binary and then run it on the node you want to join.
5654

57-
* If the Embedded Cluster Config [roles](/reference/embedded-config#roles) key is not configured, all new nodes joined to the cluster are assigned the `controller` role by default. The `controller` role designates nodes that run the Kubernetes control plane. Controller nodes can also run other workloads, such as application or Replicated KOTS workloads.
55+
1. Copy and run the tar command to extract the Embedded Cluster binary.
5856

59-
* The role cannot be changed after a node is added. If you need to change a node’s role, reset the node and add it again with the new role.
57+
1. Copy and run the join command to add the node to the cluster.
6058

61-
1. Do one of the following to make the Embedded Cluster installation assets available on the machine that you will join to the cluster:
59+
1. In the Admin Console, either on the installation **Nodes** screen or on the **Cluster Management** page, verify that the node appears. Wait for the node's status to change to Ready.
6260

63-
* **For online (internet-connected) installations**: SSH onto the machine that you will join. Then, use the same commands that you ran during installation to download and untar the Embedded Cluster installation assets on the machine. See [Online Installation with Embedded Cluster](/enterprise/installing-embedded).
61+
1. Repeat these steps for each node you want to add.
6462

65-
* **For air gap installations with limited or no outbound internet access**: On a machine that has internet access, download the Embedded Cluster installation assets (including the air gap bundle) using the same command that you ran during installation. See [Air Gap Installation with Embedded Cluster](/enterprise/installing-embedded-air-gap). Then, move the downloaded assets to the air-gapped machine that you will join, and untar.
63+
## Automate Controller Node Joins {#automate-node-joins}
6664

67-
:::important
68-
The Embedded Cluster installation assets on each node must all be the same version. If you use a different version than what is installed elsewhere in the cluster, the cluster will not be stable. To download a specific version of the Embedded Cluster assets, select a version in the **Embedded cluster install instructions** dialog.
69-
:::
65+
With Embedded Cluster, you can use the command line to get the commands for joining controller nodes, rather than having to log into the Admin Console UI to get the commands. This is especially useful when testing multi-node Embedded Cluster installations where you need to automate the process of joining controller nodes to a cluster.
7066

71-
1. On the machine that you will join to the cluster, run the join command that you copied from the Admin Console.
67+
To automate controller node joins with Embedded Cluster:
7268

73-
**Example:**
69+
1. On a node that is already joined to the cluster, run the following command:
7470

75-
```bash
76-
sudo ./APP_SLUG join 10.128.0.32:30000 TxXboDstBAamXaPdleSK7Lid
77-
```
78-
**Air Gap Example:**
71+
```bash
72+
sudo ./APP_SLUG join print-command
73+
```
74+
Where `APP_SLUG` is the unique application slug.
7975

80-
```bash
81-
sudo ./APP_SLUG join --airgap-bundle APP_SLUG.airgap 10.128.0.32:30000 TxXboDstBAamXaPdleSK7Lid
82-
```
76+
The output lists the commands for downloading the Embedded Cluster binary, extracting the binary, and joining a new controller node to the cluster.
8377

84-
1. In the Admin Console, either on the installation **Nodes** screen or on the **Cluster Management** page, verify that the node appears. Wait for the node's status to change to Ready.
78+
**Example:**
79+
80+
```
81+
sudo ./your-app-slug join print-command
82+
83+
curl -k https://172.17.0.2:30000/api/v1/embedded-cluster/binary -o embedded-cluster.tar.gz && \
84+
tar -xvf embedded-cluster.tar.gz && \
85+
sudo ./embedded-cluster join 172.17.0.2:30000 1234aBcD
86+
```
87+
**Example with JSON output:**
88+
89+
```
90+
sudo ./your-app-slug join print-command -ojson
8591
86-
1. Repeat these steps for each node you want to add.
92+
{
93+
"commands": [
94+
"curl -k https://172.17.0.2:30000/api/v1/embedded-cluster/binary -o embedded-cluster.tar.gz",
95+
"tar -xvf embedded-cluster.tar.gz",
96+
"sudo ./embedded-cluster join 172.17.0.2:30000 1234aBcD"
97+
]
98+
}
99+
```
87100

88-
## High Availability for Multi-Node Clusters (Alpha) {#ha}
101+
1. On the node that you want to join as a controller, run each of the commands provided in the `join print-command` output to download the Embedded Cluster binary, extract the binary, and join the node to the cluster.
89102

90-
:::important
91-
High availability for Embedded Cluster in an Alpha feature. This feature is subject to change, including breaking changes. For more information about this feature, reach out to Alex Parker at [alexp@replicated.com](mailto:alexp@replicated.com).
92-
:::
103+
## High Availability for Multi-Node Clusters {#ha}
93104

94105
Multi-node clusters are not highly available by default. The first node of the cluster holds important data for Kubernetes and KOTS, such that the loss of this node would be catastrophic for the cluster. Enabling high availability requires that at least three controller nodes are present in the cluster.
95106

@@ -109,13 +120,13 @@ Enabling high availability has the following requirements:
109120

110121
* The [`enable-ha`](#enable-ha-existing) command is available with Embedded Cluster 2.3.0 and later.
111122

112-
* High availability is supported only for clusters where at least three nodes with the `controller` role are present.
123+
* High availability is supported only for clusters where at least three nodes with the controller role are present.
113124

114125
### Best Practices for High Availability
115126

116127
Consider the following best practices and recommendations for creating HA clusters:
117128

118-
* At least three _controller_ nodes that run the Kubernetes control plane are required for HA. This is because clusters use a quorum system, in which more than half the nodes must be up and reachable. In clusters with three controller nodes, the Kubernetes control plane can continue to operate if one node fails because a quorum can still be reached by the remaining two nodes. By default, with Embedded Cluster, all new nodes added to a cluster are controller nodes. For information about customizing the `controller` node role, see [roles](/reference/embedded-config#roles) in _Embedded Cluster Config_.
129+
* At least three _controller_ nodes that run the Kubernetes control plane are required for HA. This is because clusters use a quorum system, in which more than half the nodes must be up and reachable. In clusters with three controller nodes, the Kubernetes control plane can continue to operate if one node fails because a quorum can still be reached by the remaining two nodes. By default, with Embedded Cluster, all new nodes added to a cluster are controller nodes. For information about customizing the controller node role, see [roles](/reference/embedded-config#roles) in _Embedded Cluster Config_.
119130

120131
* Always use an odd number of controller nodes in HA clusters. Using an odd number of controller nodes ensures that the cluster can make decisions efficiently with quorum calculations. Clusters with an odd number of controller nodes also avoid split-brain scenarios where the cluster runs as two, independent groups of nodes, resulting in inconsistencies and conflicts.
121132

@@ -129,16 +140,16 @@ To create a multi-node HA cluster:
129140

130141
1. Set up a cluster with at least two controller nodes. You can do an online (internet-connected) or air gap installation. For more information, see [Online Installation with Embedded Cluster](/enterprise/installing-embedded) or [Air Gap Installation with Embedded Cluster](/enterprise/installing-embedded-air-gap).
131142

132-
1. SSH onto a third node that you want to join to the cluster as a controller.
143+
1. Get the commands for downloading the Embedded Cluster binary, extracting the binary, and joining the third controller node either in the Admin Console **Cluster Management** tab or by running the following command on an existing node:
133144

134-
1. On the third node, run the join command provided in the Admin Console **Cluster Management** tab.
145+
```bash
146+
sudo ./APP_SLUG join print-command
147+
```
148+
Where `APP_SLUG` is the unique application slug.
135149

136-
**Example:**
150+
1. SSH onto the node that you want to add as a third controller.
137151

138-
```bash
139-
sudo ./APP_SLUG join 10.128.0.80:30000 tI13KUWITdIerfdMcWTA4Hpf
140-
```
141-
Where `APP_SLUG` is the unique slug for the application.
152+
1. On the node that you want to add as a third controller, run each of the commands that you copied.
142153

143154
:::note
144155
For Embedded Cluster versions earlier than 2.3.0, pass the `--enable-ha` flag with the `join` command.

‎docs/vendor/embedded-overview.mdx

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,6 @@ Additionally, any Helm [`extensions`](/reference/embedded-config#extensions) tha
5353

5454
### Multi-Node Architecture with High Availability
5555

56-
:::note
57-
High availability (HA) for multi-node installations with Embedded Cluster is Alpha and is not enabled by default. For more informaiton about enabling HA, see [Enable High Availability for Multi-Node Clusters (Alpha)](/enterprise/embedded-manage-nodes#ha).
58-
:::
59-
6056
<HaArchitecture/>
6157

6258
## Built-In Extensions {#built-in-extensions}
@@ -101,8 +97,6 @@ Embedded Cluster has the following limitations:
10197

10298
* **Migration from kURL**: We are helping several customers migrate from kURL to Embedded Cluster. For more information about migrating from kURL to Embedded Cluster, including key considerations before migrating and an example step-by-step migration process, see [Replicated kURL to Embedded Cluster Migration](https://docs.google.com/document/d/1Qw9owCK4xNXHRRmxDgAq_NJdxQ4O-6w2rWk_luzBD7A/edit?tab=t.0). For additional questions and to begin the migration process for your application, reach out to Alex Parker at alexp@replicated.com.
10399

104-
* **Multi-node support is in beta**: Support for multi-node embedded clusters is in beta, and enabling high availability for multi-node clusters is in alpha. Only single-node embedded clusters are generally available. For more information, see [Manage Multi-Node Clusters with Embedded Cluster](/enterprise/embedded-manage-nodes).
105-
106100
* **Disaster recovery is in alpha**: Disaster Recovery for Embedded Cluster installations is in alpha. For more information, see [Disaster Recovery for Embedded Cluster (Alpha)](/vendor/embedded-disaster-recovery).
107101

108102
* **Partial rollback support**: In Embedded Cluster 1.17.0 and later, rollbacks are supported only when rolling back to a version where there is no change to the [Embedded Cluster Config](/reference/embedded-config) compared to the currently-installed version. For example, users can roll back to release version 1.0.0 after upgrading to 1.1.0 only if both 1.0.0 and 1.1.0 use the same Embedded Cluster Config. For more information about how to enable rollbacks for your application in the KOTS Application custom resource, see [allowRollback](/reference/custom-resource-application#allowrollback) in _Application_.

‎static/images/admin-console-add-node.png

-250 KB
Binary file not shown.

0 commit comments

Comments
 (0)