You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To validate the hardware for etcd before or after you create the {product-title} cluster, you can use fio.
10
+
11
+
.Prerequisites
12
+
13
+
* Container runtimes such as Podman or Docker are installed on the machine that you are testing.
14
+
* Data is written to the `/var/lib/etcd` path.
15
+
16
+
.Procedure
17
+
* Run fio and analyze the results:
18
+
+
19
+
--
20
+
** If you use Podman, run this command:
21
+
[source,terminal]
22
+
+
23
+
----
24
+
$ sudo podman run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf
25
+
----
26
+
27
+
** If you use Docker, run this command:
28
+
[source,terminal]
29
+
+
30
+
----
31
+
$ sudo docker run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf
32
+
----
33
+
--
34
+
35
+
The output reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 10 ms. A few of the most important etcd metrics that might affected by I/O performance are as follows:
36
+
37
+
* `etcd_disk_wal_fsync_duration_seconds_bucket` metric reports the etcd's WAL fsync duration
38
+
* `etcd_disk_backend_commit_duration_seconds_bucket` metric reports the etcd backend commit latency duration
39
+
* `etcd_server_leader_changes_seen_total` metric reports the leader changes
40
+
41
+
Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed {product-title} cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.
42
+
43
+
The `histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m]))` metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.
Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance.
10
-
Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because etcd's consensus protocol depends on persistently storing metadata to a log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.
9
+
Because etcd writes data to disk and persists proposals on disk, its performance depends on disk performance. Although etcd is not particularly I/O intensive, it requires a low latency block device for optimal performance and stability. Because the consensus protocol for etcd depends on persistently storing metadata to a log (WAL), etcd is sensitive to disk-write latency. Slow disks and disk activity from other processes can cause long fsync latencies.
11
10
12
11
Those latencies can cause etcd to miss heartbeats, not commit new proposals to the disk on time, and ultimately experience request timeouts and temporary leader loss. High write latencies also lead to an OpenShift API slowness, which affects cluster performance. Because of these reasons, avoid colocating other workloads on the control-plane nodes that are I/O sensitive or intensive and share the same underlying I/O infrastructure.
13
12
14
-
In terms of latency, run etcd on top of a block device that can write at least 50 IOPS of 8000 bytes long sequentially. That is, with a latency of 10ms, keep in mind that uses fdatasync to synchronize each write in the WAL. For heavy loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as fio.
13
+
Run etcd on a block device that can write at least 50 IOPS of 8KB sequentially, including fdatasync, in under 10ms. For heavy loaded clusters, sequential 500 IOPS of 8000 bytes (2 ms) are recommended. To measure those numbers, you can use a benchmarking tool, such as fio.
15
14
16
15
To achieve such performance, run etcd on machines that are backed by SSD or NVMe disks with low latency and high throughput. Consider single-level cell (SLC) solid-state drives (SSDs), which provide 1 bit per memory cell, are durable and reliable, and are ideal for write-intensive workloads.
17
16
@@ -28,58 +27,13 @@ The following hard drive practices provide optimal etcd performance:
28
27
* Prefer high-bandwidth reads for faster recovery from failures.
29
28
* Use solid state drives as a minimum selection. Prefer NVMe drives for production environments.
30
29
* Use server-grade hardware for increased reliability.
31
-
32
-
[NOTE]
33
-
====
34
-
Avoid NAS or SAN setups and spinning drives. Ceph Rados Block Device (RBD) and other types of network-attached storage can result in unpredictable network latency. To provide fast storage to etcd nodes at scale, use PCI passthrough to pass NVM devices directly to the nodes.
35
-
====
36
-
37
-
Always benchmark by using utilities such as fio. You can use such utilities to continuously monitor the cluster performance as it increases.
38
-
39
-
[NOTE]
40
-
====
41
-
Avoid using the Network File System (NFS) protocol or other network based file systems.
42
-
====
30
+
* Avoid NAS or SAN setups and spinning drives. Ceph Rados Block Device (RBD) and other types of network-attached storage can result in unpredictable network latency. To provide fast storage to etcd nodes at scale, use PCI passthrough to pass NVM devices directly to the nodes.
31
+
* Always benchmark by using utilities such as fio. You can use such utilities to continuously monitor the cluster performance as it increases.
32
+
* Avoid using the Network File System (NFS) protocol or other network based file systems.
43
33
44
34
Some key metrics to monitor on a deployed {product-title} cluster are p99 of etcd disk write ahead log duration and the number of etcd leader changes. Use Prometheus to track these metrics.
45
35
46
36
[NOTE]
47
37
====
48
38
The etcd member database sizes can vary in a cluster during normal operations. This difference does not affect cluster upgrades, even if the leader size is different from the other members.
49
-
====
50
-
51
-
To validate the hardware for etcd before or after you create the {product-title} cluster, you can use fio.
52
-
53
-
.Prerequisites
54
-
55
-
* Container runtimes such as Podman or Docker are installed on the machine that you are testing.
56
-
* Data is written to the `/var/lib/etcd` path.
57
-
58
-
.Procedure
59
-
* Run fio and analyze the results:
60
-
+
61
-
--
62
-
** If you use Podman, run this command:
63
-
[source,terminal]
64
-
+
65
-
----
66
-
$ sudo podman run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf
67
-
----
68
-
69
-
** If you use Docker, run this command:
70
-
[source,terminal]
71
-
+
72
-
----
73
-
$ sudo docker run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/cloud-bulldozer/etcd-perf
74
-
----
75
-
--
76
-
77
-
The output reports whether the disk is fast enough to host etcd by comparing the 99th percentile of the fsync metric captured from the run to see if it is less than 10 ms. A few of the most important etcd metrics that might affected by I/O performance are as follow:
78
-
79
-
* `etcd_disk_wal_fsync_duration_seconds_bucket` metric reports the etcd's WAL fsync duration
80
-
* `etcd_disk_backend_commit_duration_seconds_bucket` metric reports the etcd backend commit latency duration
81
-
* `etcd_server_leader_changes_seen_total` metric reports the leader changes
82
-
83
-
Because etcd replicates the requests among all the members, its performance strongly depends on network input/output (I/O) latency. High network latencies result in etcd heartbeats taking longer than the election timeout, which results in leader elections that are disruptive to the cluster. A key metric to monitor on a deployed {product-title} cluster is the 99th percentile of etcd network peer latency on each etcd cluster member. Use Prometheus to track the metric.
84
-
85
-
The `histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m]))` metric reports the round trip time for etcd to finish replicating the client requests between the members. Ensure that it is less than 50 ms.
0 commit comments