Skip to content

Commit e0e92c8

Browse files
authored
Merge pull request #77561 from amolnar-rh/TELCODOCS-1707-performing-gitops
TELCODOCS-1707: Add Performing upgrade GitOps
2 parents aacb6aa + e1ed686 commit e0e92c8

File tree

6 files changed

+483
-5
lines changed

6 files changed

+483
-5
lines changed

_topic_maps/_topic_map.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3111,8 +3111,8 @@ Topics:
31113111
File: ztp-image-based-upgrade-prep-resources
31123112
- Name: Performing an image-based upgrade for single-node OpenShift clusters
31133113
File: cnf-image-based-upgrade-base
3114-
# - Name: Performing an image-based upgrade for single-node OpenShift clusters using GitOps ZTP
3115-
# File: ztp-image-based-upgrade
3114+
- Name: Performing an image-based upgrade for single-node OpenShift clusters using GitOps ZTP
3115+
File: ztp-image-based-upgrade
31163116
---
31173117
Name: Reference design specifications
31183118
Dir: telco_ref_design_specs
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="ztp-image-based-upgrade-for-sno"]
3+
= Performing an image-based upgrade for {sno} clusters using {ztp}
4+
:context: ztp-image-based-upgrade
5+
include::_attributes/common-attributes.adoc[]
6+
7+
toc::[]
8+
9+
You can upgrade your managed {sno} cluster with the image-based upgrade through {ztp-first}.
10+
11+
When you deploy the {lcao} on a cluster, an `ImageBasedUpgrade` CR is automatically created.
12+
You update this CR to specify the image repository of the seed image and to move through the different stages.
13+
14+
// Lifecycle Agent (LCAO)
15+
16+
include::modules/ztp-image-based-upgrade-talm-prep.adoc[leveloffset=+1]
17+
18+
////
19+
[role="_additional-resources"]
20+
.Additional resources
21+
22+
* xref:../../edge_computing/ztp-preparing-the-hub-cluster.adoc#ztp-preparing-the-ztp-git-repository-ver-ind_ztp-preparing-the-hub-cluster[Preparing the GitOps ZTP site configuration repository for version independence]
23+
24+
* xref:../../edge_computing/image_based_upgrade/preparing_for_image_based_upgrade/ztp-image-based-upgrade-prep-resources.adoc[Preparing for the image-based upgrade for single-node OpenShift clusters]
25+
26+
* xref:../../edge_computing/image_based_upgrade/preparing_for_image_based_upgrade/cnf-image-based-upgrade-shared-container-image.adoc#ztp-image-based-upgrade-shared-container-directory_shared-container-directory[Configuring a shared container directory between ostree stateroots when using GitOps ZTP]
27+
28+
* xref:../../backup_and_restore/application_backup_and_restore/installing/installing-oadp-ocs.adoc#oadp-about-backup-snapshot-locations_installing-oadp-ocs[About backup and snapshot locations and their secrets]
29+
30+
* xref:../../backup_and_restore/application_backup_and_restore/backing_up_and_restoring/oadp-creating-backup-cr.adoc[Creating a Backup CR]
31+
32+
* xref:../../backup_and_restore/application_backup_and_restore/backing_up_and_restoring/restoring-applications.adoc#oadp-creating-restore-cr_restoring-applications[Creating a Restore CR]
33+
////
34+
35+
include::modules/ztp-image-based-upgrade-talm-upgrade.adoc[leveloffset=+1]
36+
37+
include::modules/ztp-image-based-upgrade-talm-rollback.adoc[leveloffset=+1]
38+
39+
include::modules/cnf-image-based-upgrade-troubleshooting.adoc[leveloffset=+1]

modules/cnf-image-based-upgrade-troubleshooting.adoc

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,24 +56,29 @@ message: failed to delete all the backup CRs. Perform cleanup manually then add
5656
--
5757

5858
Resolution::
59-
59+
+
60+
--
6061
. Inspect the logs to determine why the failure occurred.
6162

6263
. To prompt {lcao} to retry the cleanup, add the `lca.openshift.io/manual-cleanup-done` annotation to the `ImageBasedUpgrade` CR.
6364

65+
+
6466
After observing this annotation, {lcao} retries the cleanup and, if it is successful, the `ImageBasedUpgrade` stage transitions to `Idle`.
6567

68+
+
6669
If the cleanup fails again, you can manually clean up the resources.
70+
--
6771

6872
[id="ztp-image-based-upgrade-troubleshooting-stateroot_{context}"]
6973
=== Cleaning up stateroot manually
7074

7175
Issue::
7276

7377
Stopping at the `Prep` stage, {lcao} cleans up the new stateroot. When finalizing after a successful upgrade or a rollback, {lcao} cleans up the old stateroot.
74-
If this step fails, it is recommended that you inspect the logs to determine why the failure occurred.
78+
If this step fails, it is recommended that you inspect the logs to determine why the failure occurred.
7579

7680
Resolution::
81+
+
7782
--
7883
. Check if there are any existing deployments in the stateroot by running the following command:
7984
+
@@ -86,21 +91,24 @@ $ ostree admin status
8691
+
8792
[source,terminal]
8893
----
89-
$ ostree admin undeploy <index_of_deployment>
94+
$ ostree admin undeploy <index_of_deployment>
9095
----
9196

9297
. After cleaning up all the deployments of the stateroot, wipe the stateroot directory by running the following commands:
9398

99+
+
94100
[WARNING]
95101
====
96102
Ensure that the booted deployment is not in this stateroot.
97103
====
98104

105+
+
99106
[source,terminal]
100107
----
101108
$ stateroot="<stateroot_to_delete>"
102109
----
103110

111+
+
104112
[source,terminal]
105113
----
106114
$ unshare -m /bin/sh -c "mount -o remount,rw /sysroot && rm -rf /sysroot/ostree/deploy/${stateroot}"
Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
// Module included in the following assemblies:
2+
// * edge_computing/image-based-upgrade/ztp-image-based-upgrade.adoc
3+
4+
:_mod-docs-content-type: PROCEDURE
5+
[id="ztp-image-based-upgrade-prep-gitops_{context}"]
6+
= Moving to the Prep stage of the image-based upgrade with {lcao} and {ztp}
7+
8+
When you deploy the {lcao} on a cluster, an `ImageBasedUpgrade` CR is automatically created. You update this CR to specify the image repository of the seed image and to move through the different stages.
9+
10+
.Prerequisites
11+
12+
* Create policies and `ConfigMap` objects for resources used in the image-based upgrade. For more information, see "Creating ConfigMap objects for the image-based upgrade with {ztp}
13+
14+
.Procedure
15+
16+
. Add policies for the `Prep`, `Upgrade`, and `Idle` stages to your existing group `PolicyGenTemplate` called `ibu-upgrade-ranGen.yaml`:
17+
+
18+
[source,yaml]
19+
----
20+
apiVersion: ran.openshift.io/v1
21+
kind: PolicyGenTemplate
22+
metadata:
23+
name: example-group-ibu
24+
namespace: "ztp-group"
25+
spec:
26+
bindingRules:
27+
group-du-sno: ""
28+
mcp: "master"
29+
evaluationInterval: <1>
30+
compliant: 10s
31+
noncompliant: 10s
32+
sourceFiles:
33+
- fileName: ConfigMapGeneric.yaml
34+
complianceType: mustonlyhave
35+
policyName: "oadp-cm-policy"
36+
metadata:
37+
name: oadp-cm
38+
namespace: openshift-adp
39+
- fileName: ibu/ImageBasedUpgrade.yaml
40+
policyName: "prep-stage-policy"
41+
spec:
42+
stage: Prep
43+
seedImageRef: <2>
44+
version: "4.15.0"
45+
image: "quay.io/user/lca-seed:4.15.0"
46+
pullSecretRef:
47+
name: "<seed_pull_secret>"
48+
oadpContent: <3>
49+
- name: "oadp-cm"
50+
namespace: "openshift-adp"
51+
status:
52+
conditions:
53+
- reason: Completed
54+
status: "True"
55+
type: PrepCompleted
56+
message: "Prep stage completed successfully"
57+
- fileName: ibu/ImageBasedUpgrade.yaml
58+
policyName: "upgrade-stage-policy"
59+
spec:
60+
stage: Upgrade
61+
status:
62+
conditions:
63+
- reason: Completed
64+
status: "True"
65+
type: UpgradeCompleted
66+
- fileName: ibu/ImageBasedUpgrade.yaml
67+
policyName: "finalize-stage-policy"
68+
complianceType: mustonlyhave
69+
spec:
70+
stage: Idle
71+
- fileName: ibu/ImageBasedUpgrade.yaml
72+
policyName: "finalize-stage-policy"
73+
status:
74+
conditions:
75+
- reason: Idle
76+
status: "True"
77+
type: Idle
78+
----
79+
<1> The policy evaluation interval for compliant and non-compliant policies. Set them to `10s` to ensure that the policies status accurately reflects the current upgrade status.
80+
<2> Define the seed image, {product-title} version, and pull secret for the upgrade in the Prep stage.
81+
<3> Define the OADP `ConfigMap` resources required for backup and restore.
82+
83+
. Verify that the policies required for an image-based upgrade are created by running the following command:
84+
+
85+
--
86+
[source,terminal]
87+
----
88+
$ oc get policies -n spoke1 | grep -E "example-group-ibu"
89+
----
90+
91+
.Example output
92+
[source,terminal]
93+
----
94+
ztp-group.example-group-ibu-oadp-cm-policy inform NonCompliant 31h
95+
ztp-group.example-group-ibu-prep-stage-policy inform NonCompliant 31h
96+
ztp-group.example-group-ibu-upgrade-stage-policy inform NonCompliant 31h
97+
ztp-group.example-group-ibu-finalize-stage-policy inform NonCompliant 31h
98+
ztp-group.example-group-ibu-rollback-stage-policy inform NonCompliant 31h
99+
----
100+
--
101+
102+
. Update the `du-profile` cluster label to the target platform version or the corresponding policy-binding label in the `SiteConfig` CR.
103+
+
104+
--
105+
[source,yaml]
106+
----
107+
apiVersion: ran.openshift.io/v1
108+
kind: SiteConfig
109+
[...]
110+
spec:
111+
[...]
112+
clusterLabels:
113+
du-profile: "4.15.0"
114+
----
115+
116+
[IMPORTANT]
117+
====
118+
Updating the labels to the target platform version unbinds the existing set of policies.
119+
====
120+
--
121+
122+
. Commit and push the updated `SiteConfig` CR to the Git repository.
123+
124+
. When you are ready to move to the `Prep` stage, create the `ClusterGroupUpgrade` CR on the target hub cluster with the `Prep` and OADP `ConfigMap` policies:
125+
+
126+
[source,yaml]
127+
----
128+
apiVersion: ran.openshift.io/v1alpha1
129+
kind: ClusterGroupUpgrade
130+
metadata:
131+
name: cgu-ibu-prep
132+
namespace: default
133+
spec:
134+
clusters:
135+
- spoke1
136+
enable: true
137+
managedPolicies:
138+
- example-group-ibu-oadp-cm-policy
139+
- example-group-ibu-prep-stage-policy
140+
remediationStrategy:
141+
canaries:
142+
- spoke1
143+
maxConcurrency: 1
144+
timeout: 240
145+
----
146+
147+
. Apply the `Prep` policy by running the following command:
148+
+
149+
--
150+
[source,terminal]
151+
----
152+
$ oc apply -f cgu-ibu-prep.yml
153+
----
154+
155+
If you provide `ConfigMap` objects for OADP resources and extra manifests, {lcao} validates the specified `ConfigMap` objects during the `Prep` stage.
156+
You might encounter the following issues:
157+
158+
* Validation warnings or errors if the {lcao} detects any issues with `extraManifests`
159+
* Validation errors if the {lcao} detects any issues with `oadpContent`
160+
161+
Validation warnings do not block the `Upgrade` stage but you must decide if it is safe to proceed with the upgrade.
162+
These warnings, for example missing CRDs, namespaces or dry run failures, update the `status.conditions` in the `Prep` stage and `annotation` fields in the `ImageBasedUpgrade` CR with details about the warning.
163+
164+
.Example validation warning
165+
[source,yaml]
166+
----
167+
[...]
168+
metadata:
169+
annotations:
170+
extra-manifest.lca.openshift.io/validation-warning: '...'
171+
[...]
172+
----
173+
174+
However, validation errors, such as adding `MachineConfig` or Operator manifests to extra manifests, cause the `Prep` stage to fail and block the `Upgrade` stage.
175+
176+
When the validations pass, the cluster creates a new `ostree` stateroot, which involves pulling and unpacking the seed image, and running host level commands.
177+
Finally, all the required images are precached on the target cluster.
178+
--
179+
180+
. Monitor the status and wait for the `cgu-ibu-prep` `ClusterGroupUpgrade` to report `Completed` by running the following command:
181+
+
182+
--
183+
[source,terminal]
184+
----
185+
$ oc get cgu -n default
186+
----
187+
188+
.Example output
189+
[source,terminal]
190+
----
191+
NAME AGE STATE DETAILS
192+
cgu-ibu-prep 31h Completed All clusters are compliant with all the managed policies
193+
----
194+
--

0 commit comments

Comments
 (0)