Skip to content

Commit fe2a3e8

Browse files
JoeAldingerJoeAldinger
authored
Merge pull request #80869 from shdeshpa07/OADP-4655-kopia-env-variables
/lgtm merging OADP-4655 Added section for kopia env variables
2 parents 2f1112c + a3076ea commit fe2a3e8

File tree

5 files changed

+431
-1
lines changed

5 files changed

+431
-1
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3377,13 +3377,15 @@ Topics:
33773377
File: oadp-using-data-mover-for-csi-snapshots-doc
33783378
- Name: Using OADP 1.2 Data Mover with Ceph storage
33793379
File: oadp-12-data-mover-ceph-doc
3380-
- Name: OADP 1.3 Data Mover
3380+
- Name: OADP 1.3 Data Mover and later
33813381
Dir: installing
33823382
Topics:
33833383
- Name: About the OADP 1.3 Data Mover
33843384
File: about-oadp-1-3-data-mover
33853385
- Name: Backing up and restoring volumes by using CSI snapshots data movement
33863386
File: oadp-backup-restore-csi-snapshots
3387+
- Name: Overriding Kopia algorithms
3388+
File: overriding-kopia-algorithms
33873389
- Name: Troubleshooting
33883390
File: troubleshooting
33893391
- Name: OADP API

‎backup_and_restore/application_backup_and_restore/installing/overriding-kopia-algorithms.adoc

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="overriding-kopia-algorithms"]
3+
= Overriding Kopia hashing, encryption, and splitter algorithms
4+
include::_attributes/common-attributes.adoc[]
5+
:context: overriding-kopia-algorithms
6+
7+
8+
toc::[]
9+
10+
You can override the default values of Kopia hashing, encryption, and splitter algorithms by using specific environment variables in the Data Protection Application (DPA).
11+
12+
// configuring the DPA with Kopia environment variables
13+
include::modules/oadp-kopia-configuring-algorithms.adoc[leveloffset=+1]
14+
// use case to demonstrate the usage of Kopia environment variables
15+
include::modules/oadp-usecase-kopia-override-algorithms.adoc[leveloffset=+1]
16+
// benchmarking Kopia algorithms
17+
include::modules/oadp-kopia-algorithms-benchmarking.adoc[leveloffset=+1]
18+

‎modules/oadp-kopia-algorithms-benchmarking.adoc

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * backup_and_restore/application_backup_and_restore/installing/overriding-kopia-algorithms.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="oadp-kopia-algorithms-benchmarking_{context}"]
7+
= Benchmarking Kopia hashing, encryption, and splitter algorithms
8+
9+
You can run Kopia commands to benchmark the hashing, encryption, and splitter algorithms. Based on the benchmarking results, you can select the most suitable algorithm for your workload. In this procedure, you run the Kopia benchmarking commands from a pod on the cluster. The benchmarking results can vary depending on CPU speed, available RAM, disk speed, current I/O load, and so on.
10+
11+
.Prerequisites
12+
13+
* You have installed the {oadp-short} Operator.
14+
* You have an application with persistent volumes running in a separate namespace.
15+
* You have run a backup of the application with Container Storage Interface (CSI) snapshots.
16+
17+
.Procedure
18+
19+
. Configure a pod as shown in the following example. Make sure you are using the `oadp-mustgather` image for {oadp-short} version 1.3 and later.
20+
+
21+
.Example pod configuration
22+
+
23+
[source,yaml]
24+
----
25+
apiVersion: v1
26+
kind: Pod
27+
metadata:
28+
name: oadp-mustgather-pod
29+
labels:
30+
purpose: user-interaction
31+
spec:
32+
containers:
33+
- name: oadp-mustgather-container
34+
image: registry.redhat.io/oadp/oadp-mustgather-rhel9:v1.3
35+
command: ["sleep"]
36+
args: ["infinity"]
37+
----
38+
+
39+
[NOTE]
40+
=====
41+
The Kopia client is available in the `oadp-mustgather` image.
42+
=====
43+
+
44+
. Create the pod by running the following command:
45+
+
46+
[source,terminal]
47+
----
48+
$ oc apply -f <pod_config_file_name> # <1>
49+
----
50+
<1> Specify the name of the YAML file for the pod configuration.
51+
+
52+
. Verify that the Security Context Constraints (SCC) on the pod is `anyuid`, so that Kopia can connect to the repository.
53+
+
54+
[source,terminal]
55+
----
56+
$ oc describe pod/oadp-mustgather-pod | grep scc
57+
----
58+
+
59+
.Example output
60+
+
61+
[source,terminal]
62+
----
63+
openshift.io/scc: anyuid
64+
----
65+
+
66+
. Connect to the pod via SSH by running the following command:
67+
+
68+
[source,terminal]
69+
----
70+
$ oc -n openshift-adp rsh pod/oadp-mustgather-pod
71+
----
72+
+
73+
. Connect to the Kopia repository by running the following command:
74+
+
75+
[source,terminal]
76+
----
77+
sh-5.1# kopia repository connect s3 \
78+
--bucket=<bucket_name> \ # <1>
79+
--prefix=velero/kopia/<application_namespace> \ # <2>
80+
--password=static-passw0rd \ # <3>
81+
--access-key="<access_key>" \ # <4>
82+
--secret-access-key="<secret_access_key>" \ # <5>
83+
--endpoint=<bucket_endpoint> \ # <6>
84+
----
85+
<1> Specify the object storage provider bucket name.
86+
<2> Specify the namespace for the application.
87+
<3> This is the Kopia password to connect to the repository.
88+
<4> Specify the object storage provider access key.
89+
<5> Specify the object storage provider secret access key.
90+
<6> Specify the bucket endpoint. You do not need to specify the bucket endpoint, if you are using {aws-short} S3 as the storage provider.
91+
+
92+
[NOTE]
93+
====
94+
This is an example command. The command can vary based on the object storage provider.
95+
====
96+
+
97+
. To benchmark the hashing algorithm, run the following command:
98+
+
99+
[source,terminal]
100+
----
101+
sh-5.1# kopia benchmark hashing
102+
----
103+
+
104+
.Example output
105+
[source,terminal]
106+
----
107+
Benchmarking hash 'BLAKE2B-256' (100 x 1048576 bytes, parallelism 1)
108+
Benchmarking hash 'BLAKE2B-256-128' (100 x 1048576 bytes, parallelism 1)
109+
Benchmarking hash 'BLAKE2S-128' (100 x 1048576 bytes, parallelism 1)
110+
Benchmarking hash 'BLAKE2S-256' (100 x 1048576 bytes, parallelism 1)
111+
Benchmarking hash 'BLAKE3-256' (100 x 1048576 bytes, parallelism 1)
112+
Benchmarking hash 'BLAKE3-256-128' (100 x 1048576 bytes, parallelism 1)
113+
Benchmarking hash 'HMAC-SHA224' (100 x 1048576 bytes, parallelism 1)
114+
Benchmarking hash 'HMAC-SHA256' (100 x 1048576 bytes, parallelism 1)
115+
Benchmarking hash 'HMAC-SHA256-128' (100 x 1048576 bytes, parallelism 1)
116+
Benchmarking hash 'HMAC-SHA3-224' (100 x 1048576 bytes, parallelism 1)
117+
Benchmarking hash 'HMAC-SHA3-256' (100 x 1048576 bytes, parallelism 1)
118+
Hash Throughput
119+
-----------------------------------------------------------------
120+
0. BLAKE3-256 15.3 GB / second
121+
1. BLAKE3-256-128 15.2 GB / second
122+
2. HMAC-SHA256-128 6.4 GB / second
123+
3. HMAC-SHA256 6.4 GB / second
124+
4. HMAC-SHA224 6.4 GB / second
125+
5. BLAKE2B-256-128 4.2 GB / second
126+
6. BLAKE2B-256 4.1 GB / second
127+
7. BLAKE2S-256 2.9 GB / second
128+
8. BLAKE2S-128 2.9 GB / second
129+
9. HMAC-SHA3-224 1.6 GB / second
130+
10. HMAC-SHA3-256 1.5 GB / second
131+
-----------------------------------------------------------------
132+
Fastest option for this machine is: --block-hash=BLAKE3-256
133+
----
134+
+
135+
. To benchmark the encryption algorithm, run the following command:
136+
+
137+
[source,terminal]
138+
----
139+
sh-5.1# kopia benchmark encryption
140+
----
141+
+
142+
.Example output
143+
[source,terminal]
144+
----
145+
Benchmarking encryption 'AES256-GCM-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1)
146+
Benchmarking encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1)
147+
Encryption Throughput
148+
-----------------------------------------------------------------
149+
0. AES256-GCM-HMAC-SHA256 2.2 GB / second
150+
1. CHACHA20-POLY1305-HMAC-SHA256 1.8 GB / second
151+
-----------------------------------------------------------------
152+
Fastest option for this machine is: --encryption=AES256-GCM-HMAC-SHA256
153+
----
154+
+
155+
. To benchmark the splitter algorithm, run the following command:
156+
+
157+
[source,terminal]
158+
----
159+
sh-5.1# kopia benchmark splitter
160+
----
161+
+
162+
.Example output
163+
[source,terminal]
164+
----
165+
splitting 16 blocks of 32MiB each, parallelism 1
166+
DYNAMIC 747.6 MB/s count:107 min:9467 10th:2277562 25th:2971794 50th:4747177 75th:7603998 90th:8388608 max:8388608
167+
DYNAMIC-128K-BUZHASH 718.5 MB/s count:3183 min:3076 10th:80896 25th:104312 50th:157621 75th:249115 90th:262144 max:262144
168+
DYNAMIC-128K-RABINKARP 164.4 MB/s count:3160 min:9667 10th:80098 25th:106626 50th:162269 75th:250655 90th:262144 max:262144
169+
# ...
170+
FIXED-512K 102.9 TB/s count:1024 min:524288 10th:524288 25th:524288 50th:524288 75th:524288 90th:524288 max:524288
171+
FIXED-8M 566.3 TB/s count:64 min:8388608 10th:8388608 25th:8388608 50th:8388608 75th:8388608 90th:8388608 max:8388608
172+
-----------------------------------------------------------------
173+
0. FIXED-8M 566.3 TB/s count:64 min:8388608 10th:8388608 25th:8388608 50th:8388608 75th:8388608 90th:8388608 max:8388608
174+
1. FIXED-4M 425.8 TB/s count:128 min:4194304 10th:4194304 25th:4194304 50th:4194304 75th:4194304 90th:4194304 max:4194304
175+
# ...
176+
22. DYNAMIC-128K-RABINKARP 164.4 MB/s count:3160 min:9667 10th:80098 25th:106626 50th:162269 75th:250655 90th:262144 max:262144
177+
----
178+

‎modules/oadp-kopia-configuring-algorithms.adoc

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * backup_and_restore/application_backup_and_restore/installing/overriding-kopia-algorithms.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="oadp-kopia-configuring-algorithms_{context}"]
7+
= Configuring the DPA to override Kopia hashing, encryption, and splitter algorithms
8+
9+
You can use an {oadp-first} option to override the default Kopia algorithms for hashing, encryption, and splitter to improve Kopia performance or to compare performance metrics. You can set the following environment variables in the `spec.configuration.velero.podConfig.env` section of the DPA:
10+
11+
* `KOPIA_HASHING_ALGORITHM`
12+
* `KOPIA_ENCRYPTION_ALGORITHM`
13+
* `KOPIA_SPLITTER_ALGORITHM`
14+
15+
.Prerequisites
16+
17+
* You have installed the {oadp-short} Operator.
18+
* You have created the secret by using the credentials provided by the cloud provider.
19+
20+
.Procedure
21+
22+
* Configure the DPA with the environment variables for hashing, encryption, and splitter as shown in the following example.
23+
+
24+
.Example DPA
25+
[source,yaml]
26+
----
27+
apiVersion: oadp.openshift.io/v1alpha1
28+
kind: DataProtectionApplication
29+
#...
30+
configuration:
31+
nodeAgent:
32+
enable: true # <1>
33+
uploaderType: kopia # <2>
34+
velero:
35+
defaultPlugins:
36+
- openshift
37+
- aws
38+
- csi #<3>
39+
defaultSnapshotMoveData: true
40+
podConfig:
41+
env:
42+
- name: KOPIA_HASHING_ALGORITHM
43+
value: <hashing_algorithm_name> # <4>
44+
- name: KOPIA_ENCRYPTION_ALGORITHM
45+
value: <encryption_algorithm_name> # <5>
46+
- name: KOPIA_SPLITTER_ALGORITHM
47+
value: <splitter_algorithm_name> # <6>
48+
----
49+
<1> Enable the `nodeAgent`.
50+
<2> Specify the `uploaderType` as `kopia`.
51+
<3> Include the `csi` plugin.
52+
<4> Specify a hashing algorithm. For example, `BLAKE3-256`.
53+
<5> Specify an encryption algorithm. For example, `CHACHA20-POLY1305-HMAC-SHA256`.
54+
<6> Specify a splitter algorithm. For example, `DYNAMIC-8M-RABINKARP`.

0 commit comments

Comments
 (0)