Skip to content

Commit 25a77b4

Browse files
bscott-rhbscott-rh
authored
Merge pull request #76718 from lahinson/osdocs-10057-etcd-db-sizing
[OSDOCS-10057]: Add TP docs about increasing DB size for etcd
2 parents b8397f3 + 747e57f commit 25a77b4

File tree

2 files changed

+196
-1
lines changed

2 files changed

+196
-1
lines changed

‎modules/etcd-increase-db.adoc

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * scalability_and_performance/recommended-performance-scale-practices/recommended-etcd-practices.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="etcd-increase-db_{context}"]
7+
= Increasing the database size for etcd
8+
9+
You can set the disk quota in gibibytes (GiB) for each etcd instance. If you set a disk quota for your etcd instance, you can specify integer values from 8 to 32. The default value is 8. You can specify only increasing values.
10+
11+
You might want to increase the disk quota if you encounter a `low space` alert. This alert indicates that the cluster is too large to fit in etcd despite automatic compaction and defragmentation. If you see this alert, you need to increase the disk quota immediately because after etcd runs out of space, writes fail.
12+
13+
Another scenario where you might want to increase the disk quota is if you encounter an `excessive database growth` alert. This alert is a warning that the database might grow too large in the next four hours. In this scenario, consider increasing the disk quota so that you do not eventually encounter a `low space` alert and possible write fails.
14+
15+
When you increase the disk quota, the amount of space that you specify is not immediately reserved. Instead, etcd can grow to that size if needed. Ensure that etcd is running on a dedicated disk that is larger than the value that you specify for the disk quota.
16+
17+
:FeatureName: Increasing the database size for etcd
18+
include::snippets/technology-preview.adoc[]
19+
20+
[id="etcd-change-db-size_{context}"]
21+
== Changing the etcd database size
22+
23+
To change the database size for etcd, complete the following steps.
24+
25+
.Procedure
26+
27+
. Check the current value of the disk quota for each etcd instance by entering the following command:
28+
+
29+
[source,terminal]
30+
----
31+
$ oc describe etcd/cluster | grep "Backend Quota"
32+
----
33+
+
34+
.Example output
35+
[source,terminal]
36+
----
37+
Backend Quota Gi B: <value>
38+
----
39+
40+
. Change the value of the disk quota by entering the following command:
41+
+
42+
[source,terminal]
43+
----
44+
$ oc patch etcd/cluster --type=merge -p '{"spec": {"backendQuotaGiB": <value>}}'
45+
----
46+
+
47+
.Example output
48+
[source,terminal]
49+
----
50+
etcd.operator.openshift.io/cluster patched
51+
----
52+
53+
.Verification
54+
55+
. Verify that the new value for the disk quota is set by entering the following command:
56+
+
57+
[source,terminal]
58+
----
59+
$ oc describe etcd/cluster | grep "Backend Quota"
60+
----
61+
+
62+
The etcd Operator automatically rolls out the etcd instances with the new values.
63+
64+
. Verify that the etcd pods are up and running by entering the following command:
65+
+
66+
[source,terminal]
67+
----
68+
oc get pods -n openshift-etcd
69+
----
70+
+
71+
The following output shows the expected entries.
72+
+
73+
.Example output
74+
[source,terminal]
75+
----
76+
NAME READY STATUS RESTARTS AGE
77+
etcd-ci-ln-b6kfsw2-72292-mzwbq-master-0 4/4 Running 0 39m
78+
etcd-ci-ln-b6kfsw2-72292-mzwbq-master-1 4/4 Running 0 37m
79+
etcd-ci-ln-b6kfsw2-72292-mzwbq-master-2 4/4 Running 0 41m
80+
etcd-guard-ci-ln-b6kfsw2-72292-mzwbq-master-0 1/1 Running 0 51m
81+
etcd-guard-ci-ln-b6kfsw2-72292-mzwbq-master-1 1/1 Running 0 49m
82+
etcd-guard-ci-ln-b6kfsw2-72292-mzwbq-master-2 1/1 Running 0 54m
83+
installer-5-ci-ln-b6kfsw2-72292-mzwbq-master-1 0/1 Completed 0 51m
84+
installer-7-ci-ln-b6kfsw2-72292-mzwbq-master-0 0/1 Completed 0 46m
85+
installer-7-ci-ln-b6kfsw2-72292-mzwbq-master-1 0/1 Completed 0 44m
86+
installer-7-ci-ln-b6kfsw2-72292-mzwbq-master-2 0/1 Completed 0 49m
87+
installer-8-ci-ln-b6kfsw2-72292-mzwbq-master-0 0/1 Completed 0 40m
88+
installer-8-ci-ln-b6kfsw2-72292-mzwbq-master-1 0/1 Completed 0 38m
89+
installer-8-ci-ln-b6kfsw2-72292-mzwbq-master-2 0/1 Completed 0 42m
90+
revision-pruner-7-ci-ln-b6kfsw2-72292-mzwbq-master-0 0/1 Completed 0 43m
91+
revision-pruner-7-ci-ln-b6kfsw2-72292-mzwbq-master-1 0/1 Completed 0 43m
92+
revision-pruner-7-ci-ln-b6kfsw2-72292-mzwbq-master-2 0/1 Completed 0 43m
93+
revision-pruner-8-ci-ln-b6kfsw2-72292-mzwbq-master-0 0/1 Completed 0 42m
94+
revision-pruner-8-ci-ln-b6kfsw2-72292-mzwbq-master-1 0/1 Completed 0 42m
95+
revision-pruner-8-ci-ln-b6kfsw2-72292-mzwbq-master-2 0/1 Completed 0 42m
96+
----
97+
98+
. Verify that the disk quota value is updated for the etcd pod by entering the following command:
99+
+
100+
[source,terminal]
101+
----
102+
$ oc describe -n openshift-etcd pod/<etcd_podname> | grep "ETCD_QUOTA_BACKEND_BYTES"
103+
----
104+
+
105+
The value might not have changed from the default value of `8`.
106+
+
107+
.Example output
108+
[source,terminal]
109+
----
110+
ETCD_QUOTA_BACKEND_BYTES: 8589934592
111+
----
112+
+
113+
[NOTE]
114+
=====
115+
While the value that you set is an integer in GiB, the value shown in the output is converted to bytes.
116+
=====
117+
118+
119+
[id="etcd-change-db-size-troubleshooting_{context}"]
120+
== Troubleshooting
121+
122+
If you encounter issues when you try to increase the database size for etcd, the following troubleshooting steps might help.
123+
124+
[id="etcd-troubleshoot-small-value_{context}"]
125+
=== Value is too small
126+
127+
If the value that you specify is less than `8`, you see the following error message:
128+
129+
[source,terminal]
130+
----
131+
$ oc patch etcd/cluster --type=merge -p '{"spec": {"backendQuotaGiB": 5}}'
132+
----
133+
134+
.Example error message
135+
[source,terminal]
136+
----
137+
The Etcd "cluster" is invalid:
138+
* spec.backendQuotaGiB: Invalid value: 5: spec.backendQuotaGiB in body should be greater than or equal to 8
139+
* spec.backendQuotaGiB: Invalid value: "integer": etcd backendQuotaGiB may not be decreased
140+
----
141+
142+
To resolve this issue, specify an integer between `8` and `32`.
143+
144+
[id="etcd-troubleshoot-large-value_{context}"]
145+
=== Value is too large
146+
147+
If the value that you specify is greater than `32`, you see the following error message:
148+
149+
[source,terminal]
150+
----
151+
$ oc patch etcd/cluster --type=merge -p '{"spec": {"backendQuotaGiB": 64}}'
152+
----
153+
154+
.Example error message
155+
[source,terminal]
156+
----
157+
The Etcd "cluster" is invalid: spec.backendQuotaGiB: Invalid value: 64: spec.backendQuotaGiB in body should be less than or equal to 32
158+
----
159+
160+
To resolve this issue, specify an integer between `8` and `32`.
161+
162+
[id="etcd-troubleshoot-decreasing-value_{context}"]
163+
=== Value is decreasing
164+
165+
If the value is set to a valid value between `8` and `32`, you cannot decrease the value. Otherwise, you see an error message.
166+
167+
. Check to see the current value by entering the following command:
168+
+
169+
[source,terminal]
170+
----
171+
$ oc describe etcd/cluster | grep "Backend Quota"
172+
----
173+
+
174+
.Example output
175+
[source,terminal]
176+
----
177+
Backend Quota Gi B: 10
178+
----
179+
180+
. Decrease the disk quota value by entering the following command:
181+
+
182+
[source,terminal]
183+
----
184+
$ oc patch etcd/cluster --type=merge -p '{"spec": {"backendQuotaGiB": 8}}'
185+
----
186+
+
187+
.Example error message
188+
[source,terminal]
189+
----
190+
The Etcd "cluster" is invalid: spec.backendQuotaGiB: Invalid value: "integer": etcd backendQuotaGiB may not be decreased
191+
----
192+
193+
. To resolve this issue, specify an integer greater than `10`.

‎scalability_and_performance/recommended-performance-scale-practices/recommended-etcd-practices.adoc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,6 @@ include::modules/etcd-tuning-parameters.adoc[leveloffset=+1]
2727

2828
[role="_additional-resources"]
2929
.Additional resources
30-
xref:../../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling-features-about_nodes-cluster-enabling[Understanding feature gates]
30+
xref:../../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling-features-about_nodes-cluster-enabling[Understanding feature gates]
31+
32+
include::modules/etcd-increase-db.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)