Skip to content

Commit de09779

Browse files
author
Nils Koenig
committed
Updated sap_hypervisor_node_preconfigure(redhat_ocpv)
 - sap_hypervisor_node_preconfigure(redhat_ocpv) - Added assert for kubeconfig - Unified storageclass to sapstorage for both trident and hpp and make configurable - trident: make parameters configurable in role - removed unused create-sap-bridge.yml, label-worker-invtsc.yml, sriov-enabled-unsupported-nics.sh - restructured tasks - added flags for finegranular tasks control, mainly for debugging - add vlan interface support - nmstate: wait for webhooks pods to be available - fix bridge and NAD name for additional bridges - added MCP wait to when installing hostpath provisioner - Storageclass names and default is configurable now - Wait and check for hco-webhook pod - hostpath provisioner: mkfs.xfs in pod, simplified systemd startup scripts
1 parent 8e656d2 commit de09779

26 files changed

+341
-365
lines changed

‎playbooks/sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml

Lines changed: 15 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -6,91 +6,34 @@
66
sap_hypervisor_node_preconfigure_platform: redhat_ocp_virt
77
tasks:
88

9-
- name: Use kubeconfig file specified in environment variable K8S_AUTH_KUBECONFIG if sap_hypervisor_node_preconfigure_ocp_kubeconfig_path is not defined
9+
- name: Use kubeconfig file specified in environment variable K8S_AUTH_KUBECONFIG | KUBECONFIG if sap_hypervisor_node_preconfigure_ocp_kubeconfig_path is not defined.
1010
when: >
11-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_path is not defined or
12-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_path == None or
13-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_path == ''
11+
sap_hypervisor_node_preconfigure_kubeconfig is not defined or
12+
sap_hypervisor_node_preconfigure_kubeconfig == None or
13+
sap_hypervisor_node_preconfigure_kubeconfig == ''
1414
ansible.builtin.set_fact:
15-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_path: "{{ lookup('env', 'K8S_AUTH_KUBECONFIG') | default(None) }}"
15+
sap_hypervisor_node_preconfigure_kubeconfig: "{{ lookup('env', 'K8S_AUTH_KUBECONFIG') | default(lookup('env', 'KUBECONFIG'), true) }}"
16+
17+
- name: Ensure that kubeconfig is set
18+
assert:
19+
that:
20+
- sap_hypervisor_node_preconfigure_kubeconfig is defined
21+
- sap_hypervisor_node_preconfigure_kubeconfig is not none
22+
- sap_hypervisor_node_preconfigure_kubeconfig | length > 0
23+
fail_msg: "sap_hypervisor_node_preconfigure_kubeconfig is required."
1624

1725
- name: Create Tempdir on jumphost
1826
ansible.builtin.tempfile:
1927
state: directory
2028
suffix: "_sap_hypervisor_node_preconfigure"
2129
register: __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost
2230

23-
- name: Set kubeconfig file variable
24-
ansible.builtin.set_fact:
25-
__sap_hypervisor_node_preconfigure_register_kubeconfig: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost.path }}/kubeconfig"
26-
27-
- name: Read content of kubeconfig file
28-
ansible.builtin.set_fact:
29-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_data:
30-
"{{ lookup('file', sap_hypervisor_node_preconfigure_ocp_kubeconfig_path) | from_yaml }}"
31-
32-
- name: Read cluster endpoint and CA certificate from kubeconfig if either is not defined
33-
when: sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig
34-
block:
35-
36-
- name: Set sap_hypervisor_node_preconfigure_ocp_endpoint from kubeconfig
37-
ansible.builtin.set_fact:
38-
__sap_hypervisor_node_preconfigure_register_ocp_endpoint:
39-
"{{ sap_hypervisor_node_preconfigure_ocp_kubeconfig_data['clusters'][0]['cluster']['server'] }}"
40-
41-
- name: Write the certificate-authority-data to temp dir on jumphost
42-
ansible.builtin.copy:
43-
content: "{{ sap_hypervisor_node_preconfigure_ocp_kubeconfig_data['clusters'][0]['cluster']['certificate-authority-data'] | b64decode }}"
44-
dest: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost.path }}/cluster-ca-cert.pem"
45-
mode: "0666"
46-
47-
- name: Set CA file variable
48-
ansible.builtin.set_fact:
49-
__sap_hypervisor_node_preconfigure_register_ca_cert: "{{ __sap_hypervisor_node_preconfigure_register_tmpdir_jumphost.path }}/cluster-ca-cert.pem"
50-
51-
52-
- name: Use predefined CA cert and API endpoint
53-
when: not sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig
54-
55-
block:
56-
- name: Set predefined OCP API Endpoint
57-
ansible.builtin.set_fact:
58-
__sap_hypervisor_node_preconfigure_register_ocp_endpoint: "{{ sap_hypervisor_node_preconfigure_ocp_endpoint }}"
59-
60-
- name: Set predefined CA file
61-
ansible.builtin.set_fact:
62-
__sap_hypervisor_node_preconfigure_register_ca_cert: "{{ sap_hypervisor_node_preconfigure_ocp_ca_cert }}"
63-
64-
65-
- name: Log into Red Hat OpenShift cluster (obtain access token)
66-
community.okd.openshift_auth:
67-
host: "{{ __sap_hypervisor_node_preconfigure_register_ocp_endpoint }}"
68-
username: "{{ sap_hypervisor_node_preconfigure_ocp_admin_username }}"
69-
password: "{{ sap_hypervisor_node_preconfigure_ocp_admin_password }}"
70-
ca_cert: "{{ __sap_hypervisor_node_preconfigure_register_ca_cert }}"
71-
register: __sap_vm_provision_register_ocp_auth_results
72-
73-
- name: Set token in kubeconfig
74-
ansible.builtin.set_fact:
75-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_data: >-
76-
{{
77-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_data | combine({
78-
'users': sap_hypervisor_node_preconfigure_ocp_kubeconfig_data.users | map('combine', [{'user': {'token': __sap_vm_provision_register_ocp_auth_results.openshift_auth.api_key }}] )
79-
}, recursive=True)
80-
}}
81-
82-
- name: Write the updated kubeconfig
83-
ansible.builtin.copy:
84-
content: "{{ sap_hypervisor_node_preconfigure_ocp_kubeconfig_data | to_nice_yaml }}"
85-
dest: "{{ __sap_hypervisor_node_preconfigure_register_kubeconfig }}"
86-
mode: "0600"
87-
8831
- name: Invoke role with credentials set as environment variables
8932
delegate_to: "{{ inventory_hostname }}"
9033
delegate_facts: true
9134
environment:
92-
KUBECONFIG: "{{ __sap_hypervisor_node_preconfigure_register_kubeconfig }}"
93-
K8S_AUTH_KUBECONFIG: "{{ __sap_hypervisor_node_preconfigure_register_kubeconfig }}"
35+
KUBECONFIG: "{{ sap_hypervisor_node_preconfigure_kubeconfig }}"
36+
K8S_AUTH_KUBECONFIG: "{{ sap_hypervisor_node_preconfigure_kubeconfig }}"
9437
block:
9538

9639
- name: Include sap_hypervisor_node_preconfigure Ansible Role

‎playbooks/vars/sample-variables-sap-hypervisor-redhat-ocp-virt-preconfigure.yml

Lines changed: 105 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -6,28 +6,11 @@
66
# Red Hat OpenShift cluster connection details
77
###########################################################
88

9-
# Admin username for Red Hat OpenShift cluster connection
10-
sap_hypervisor_node_preconfigure_ocp_admin_username:
11-
12-
# Admin password for Red Hat OpenShift cluster connection
13-
sap_hypervisor_node_preconfigure_ocp_admin_password:
14-
15-
# Path to kubeconfig file Red Hat OpenShift cluster connection
16-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_path:
17-
18-
# If this is set to true, the API endpoint and the
19-
# CA Certificate are extracted from the kubeconfig file.
20-
# If set to false, sap_hypervisor_node_preconfigure_ocp_endpoint and
21-
# sap_hypervisor_node_preconfigure_ocp_ca_cert have to be specified.
22-
sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig: true
23-
24-
# URL to the API endpoint of Red Hat OpenShift cluster
25-
#sap_hypervisor_node_preconfigure_ocp_endpoint:
26-
27-
# CA Certificate for Red Hat OpenShift cluster connection
28-
# To extract the CA Cert from the kubeconfig, you can use
29-
# grep certificate-authority-data ${KUBECONFIG} | awk '{ print $2 }' | base64 --decode > client-cert.pem
30-
#sap_hypervisor_node_preconfigure_ocp_ca_cert:
9+
# kubeconfig for Red Hat OpenShift cluster connection.
10+
# Needs to contain a valid API token for trident storage operator to work.
11+
# If not provided, the kubeconfig will be read from the environment variables
12+
# KUBECONFIG or K8S_AUTH_KUBECONFIG
13+
# sap_hypervisor_node_preconfigure_kubeconfig:
3114

3215

3316
###########################################################
@@ -47,6 +30,21 @@ sap_hypervisor_node_preconfigure_install_operators: true
4730
sap_hypervisor_node_preconfigure_setup_worker_nodes: true
4831

4932

33+
###################################################################################
34+
# Individual worker nodes config details, typically no need to change anything here.
35+
# Requires sap_hypervisor_node_preconfigure_setup_worker_nodes to be set to true.
36+
###################################################################################
37+
38+
# Should the network of the nodes be configured?
39+
sap_hypervisor_node_preconfigure_configure_worker_node_network: true
40+
41+
# Should the cpumanager be enabled? This has to be set to true.
42+
sap_hypervisor_node_preconfigure_enable_cpumanager: true
43+
44+
# Should the kargs be applied to the worker nodes?
45+
sap_hypervisor_node_preconfigure_enable_kargs: true
46+
47+
5048
###########################################################
5149
# Configuration details
5250
###########################################################
@@ -69,8 +67,11 @@ sap_hypervisor_node_preconfigure_ignore_minimal_memory_check: false
6967
# Namespace to be used for the Red Hat Openshift Virtualization Operator
7068
sap_hypervisor_node_preconfigure_ocpv_namespace: openshift-cnv
7169

72-
# Channel to be used for the Red Hat Openshift Virtualization Operator
73-
sap_hypervisor_node_preconfigure_ocpv_subscription_channel: stable
70+
# Namespace to be used for the Red Hat Openshift Nmstate Operator
71+
sap_hypervisor_node_preconfigure_nmstate_namespace: openshift-nmstate
72+
73+
# Mountpoint for hostpath provisioner
74+
sap_hypervisor_node_preconfigure_hpp_mountpoint: /var/localstorage
7475

7576

7677
###########################################################
@@ -96,13 +97,23 @@ sap_hypervisor_node_preconfigure_cluster_config:
9697
password: xxxxx
9798
storage_driver: ontap-nas
9899
storage_prefix: ocpv_sap_
100+
nfs_mount_options: nfsvers=3
101+
export_policy: default
102+
storageclass_name: sapstorage # Name of the storage class (careful, there can only be one SC with this name)
103+
default_storageclass: True # Should this storage class be default?
99104

100105
# CPU cores which will be reserved for kubernetes
101106
worker_kubernetes_reserved_cpus: "0,1"
102107

103108
# Storage device used for host path provisioner as local storage.
104109
worker_localstorage_device: /dev/vdb
105110

111+
# Name of the storage class for locastorage
112+
worker_localstorage_storageclass_name: sapstorage
113+
114+
# Should the local storage class be default?
115+
worker_localstorage_default_storageclass: True
116+
106117
# detailed configuration for every worker that should be configured
107118
workers:
108119

@@ -124,39 +135,41 @@ sap_hypervisor_node_preconfigure_cluster_config:
124135
port:
125136
- name: ens1f0 # network IF name
126137

127-
- name: storage # an SR-IOV device
128-
interface: ens2f0 # network IF name
129-
type: sriov
130-
131-
- name: ens2f0 # Set elevated MTU of 9000
132-
type: ethernet # on parent interface of
133-
state: up # storagebridge
134-
ipv4:
135-
dhcp: false
136-
enabled: false
137-
mtu: 9000
138-
139-
- name: storagebridge # using a bridge
140-
bridge: # another bridge
141-
options:
142-
stp:
143-
enabled: false
144-
port:
145-
- name: ens2f0 # network IF name
146-
description: storage
147-
mtu: 9000
148-
ipv4:
149-
address:
150-
- ip: 192.168.1.10 # IP config
151-
prefix-length: 24
152-
auto-dns: false
153-
auto-gateway: false
154-
state: up
155-
type: linux-bridge
156-
157-
- name: multi # another SR-IOV device
158-
interface: ens2f1 # network IF name
159-
type: sriov
138+
# Additional network interfaces, just shown here for reference
139+
# - name: storage # an SR-IOV device
140+
# interface: ens2f0 # network IF name
141+
# type: sriov
142+
143+
# - name: ens2f0 # Set elevated MTU of 9000
144+
# type: ethernet # on parent interface of
145+
# state: up # storagebridge
146+
# ipv4:
147+
# dhcp: false
148+
# enabled: false
149+
# mtu: 9000
150+
151+
# - name: storagebridge # using a bridge
152+
# bridge: # another bridge
153+
# options:
154+
# stp:
155+
# enabled: false
156+
# port:
157+
# - name: ens2f0 # network IF name
158+
# description: storage
159+
# mtu: 9000
160+
# ipv4:
161+
# address:
162+
# - ip: 192.168.1.10 # IP config
163+
# prefix-length: 24
164+
# auto-dns: false
165+
# auto-gateway: false
166+
# enabled: true
167+
# state: up
168+
# type: linux-bridge
169+
#
170+
# - name: multi # another SR-IOV device
171+
# interface: ens2f1 # network IF name
172+
# type: sriov
160173

161174

162175
- name: worker-1 # second worker configuration
@@ -177,36 +190,38 @@ sap_hypervisor_node_preconfigure_cluster_config:
177190
port:
178191
- name: ens1f0 # network IF name
179192

180-
- name: storage # an SR-IOV device
181-
interface: ens2f0 # network IF name
182-
type: sriov
183-
184-
- name: ens2f0 # Set elevated MTU of 9000
185-
type: ethernet # on parent interface of
186-
state: up # storagebridge
187-
ipv4:
188-
dhcp: false
189-
enabled: false
190-
mtu: 9000
191-
192-
- name: storagebridge # create storage bridge
193-
bridge:
194-
options:
195-
stp:
196-
enabled: false
197-
port:
198-
- name: ens2f0 # network IF name
199-
description: storage
200-
mtu: 9000
201-
ipv4:
202-
address:
203-
- ip: 192.168.1.11 # IP config
204-
prefix-length: 24
205-
auto-dns: false
206-
auto-gateway: false
207-
state: up
208-
type: linux-bridge
209-
210-
- name: multi # another SR-IOV device
211-
interface: ens2f1 # network IF name
212-
type: sriov
193+
# Additional network interfaces, just shown here for reference
194+
# - name: storage # an SR-IOV device
195+
# interface: ens2f0 # network IF name
196+
# type: sriov
197+
#
198+
# - name: ens2f0 # Set elevated MTU of 9000
199+
# type: ethernet # on parent interface of
200+
# state: up # storagebridge
201+
# ipv4:
202+
# dhcp: false
203+
# enabled: false
204+
# mtu: 9000
205+
#
206+
# - name: storagebridge # create storage bridge
207+
# bridge:
208+
# options:
209+
# stp:
210+
# enabled: false
211+
# port:
212+
# - name: ens2f0 # network IF name
213+
# description: storage
214+
# mtu: 9000
215+
# ipv4:
216+
# address:
217+
# - ip: 192.168.1.11 # IP config
218+
# prefix-length: 24
219+
# auto-dns: false
220+
# auto-gateway: false
221+
# enabled: true
222+
# state: up
223+
# type: linux-bridge
224+
#
225+
# - name: multi # another SR-IOV device
226+
# interface: ens2f1 # network IF name
227+
# type: sriov

‎roles/sap_hypervisor_node_preconfigure/README.md

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -103,23 +103,14 @@ Let's have a look at the most important variables you need to set.
103103
# Red Hat OpenShift cluster connection details
104104
###########################################################
105105
106-
# Admin username for Red Hat OpenShift cluster connection
107-
sap_hypervisor_node_preconfigure_ocp_admin_username:
108-
109-
# Admin password for Red Hat OpenShift cluster connection
110-
sap_hypervisor_node_preconfigure_ocp_admin_password:
111-
112-
# Path to kubeconfig file Red Hat OpenShift cluster connection
113-
sap_hypervisor_node_preconfigure_ocp_kubeconfig_path:
114-
115-
# If this is set to true, the API endpoint and the
116-
# CA Certificate are extracted from the kubeconfig file.
117-
# If set to false, sap_hypervisor_node_preconfigure_ocp_endpoint and
118-
# sap_hypervisor_node_preconfigure_ocp_ca_cert have to be specified.
119-
sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig: true
106+
# kubeconfig file Red Hat OpenShift cluster connection.
107+
# Needs to contain a valid API token for trident storage operator to work.
108+
# If not provided, the kubeconfig will be read from the environment variables
109+
# KUBECONFIG or K8S_AUTH_KUBECONFIG
110+
sap_hypervisor_node_preconfigure_kubeconfig:
120111
121112
```
122-
You need to provide username and password for the Red Hat OpenShift Cluster. The `kubeconfig` file can be specified in `sap_hypervisor_node_preconfigure_ocp_kubeconfig_path` or if omitted, the environment variable `K8S_AUTH_KUBECONFIG` has to point to it. Default is, to use the CA certificate and Red Hat OpenShift cluster API endpoint as specified in the `kubeconfig` file (controlled by variable `sap_hypervisor_node_preconfigure_ocp_extract_kubeconfig`). Make sure to specify the username and password for the cluster: `sap_hypervisor_node_preconfigure_ocp_admin_username` and `sap_hypervisor_node_preconfigure_ocp_admin_password`.
113+
You need to provide a `kubeconfig` file in `sap_hypervisor_node_kubeconfig` or if omitted, either the environment variables `K8S_AUTH_KUBECONFIG` or `KUBECONFIG` has to point to it. If using the trident storage operator, the `kubeconfig` has also to contain a valid API token.
123114

124115
Next are variables that define what storage configuration should be configured, if the operators should be installed and the configuration of the workers should be done.
125116

‎roles/sap_hypervisor_node_preconfigure/tasks/platform/redhat_ocp_virt/cluster/enable-featuregate-downwardmetrics.yml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
- name: Enable DownwardMetrics Feature Gate on kubevirt-hyperconverged
3+
kubernetes.core.k8s:
4+
state: present
5+
definition:
6+
apiVersion: hco.kubevirt.io/v1beta1
7+
kind: HyperConverged
8+
metadata:
9+
name: kubevirt-hyperconverged
10+
namespace: "{{ sap_hypervisor_node_preconfigure_ocpv_namespace }}"
11+
spec:
12+
featureGates:
13+
downwardMetrics: true

0 commit comments

Comments
 (0)