Skip to content

Commit df70980

Browse files
ShaunaDiazShaunaDiaz
authored
Merge pull request #74615 from ShaunaDiaz/OSDOCS-9812
OSDOCS-9812: add custom CA certs MicroShift
2 parents a1bfa14 + b571634 commit df70980

File tree

6 files changed

+273
-0
lines changed

6 files changed

+273
-0
lines changed

‎_topic_maps/_topic_map_ms.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -393,6 +393,8 @@ Topics:
393393
File: microshift-using-config-tools
394394
- Name: Cluster access with kubeconfig
395395
File: microshift-cluster-access-kubeconfig
396+
- Name: Using custom certificate authorities
397+
File: microshift-custom-ca
396398
- Name: Checking the status of Greenboot health checks
397399
File: microshift-greenboot-checking-status
398400
---

‎microshift_configuring/microshift-custom-ca.adoc

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="microshift-custom-ca"]
3+
= Configuring custom certificate authorities
4+
include::_attributes/attributes-microshift.adoc[]
5+
:context: microshift-custom-ca
6+
7+
toc::[]
8+
9+
You can encrypt connections by using custom certificate authorities (CAs) with the {microshift-short} service.
10+
11+
include::modules/microshift-custom-ca-con.adoc[leveloffset=+1]
12+
13+
include::modules/microshift-custom-ca-proc.adoc[leveloffset=+1]
14+
15+
include::modules/microshift-custom-ca-reserved-names.adoc[leveloffset=+1]
16+
17+
include::modules/microshift-custom-ca-troubleshooting.adoc[leveloffset=+1]
18+
19+
[id="Additional-resources_microshift-custom-ca_{context}"]
20+
== Additional resources
21+
* link:https://docs.openshift.com/container-platform/{ocp-version}/security/certificates/api-server.html#customize-certificates-api-add-named_api-server-certificates[OpenShift: Add an API server named certificate]
22+
23+
* link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/securing_networks/creating-and-managing-tls-keys-and-certificates_securing-networks#doc-wrapper[RHEL: Creating and managing TLS keys and certificates]
24+
25+
* link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/securing_networks/using-shared-system-certificates_securing-networks#the-system-wide-trust-store_using-shared-system-certificates[The system-wide truststore] for details.
26+
27+
* link:https://docs.openshift.com/container-platform/{ocp-version}/cli_reference/openshift_cli/managing-cli-profiles.html[OpenShift CLI Reference: oc login]

‎modules/microshift-custom-ca-con.adoc

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * microshift_security_compliance/microshift-custom-ca.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="microshift-custom-cas_{context}"]
7+
= How custom certificate authorities work in {microshift-short}
8+
9+
The default API server certificate is issued by an internal {microshift-short} cluster certificate authority (CA). Clients outside of the cluster cannot verify the API server certificate by default. This certificate can be replaced by a custom server certificate that is issued externally by a custom CA that clients trust. The following steps illustrate the workflow in {microshift-short}:
10+
11+
. Copy the certificates and keys to the preferred directory in the host operating system. Ensure that the files are accessible by root only.
12+
13+
. Update the {microshift-short} configuration for each custom CA by specifying the certificate names and new fully qualified domain name (FQDN) in the {microshift-short} `/etc/microshift/config.yaml` configuration file.
14+
+
15+
Each certificate configuration can contain the following values:
16+
17+
** The certificate file location is a required value.
18+
** A single common name containing the API server DNS and IP address or IP address range.
19+
+
20+
--
21+
[TIP]
22+
====
23+
In most cases, {microshift-short} generates a new `kubeconfig` for your custom CA that includes the IP address or range that you specify. The exception is when wildcards are specified for the IP address. In this case, {microshift-short} generates a `kubeconfig` with the public IP address of the server. To use wildcards, you must update the `kubeconfig` file with your specific details.
24+
====
25+
--
26+
** Multiple Subject Alternative Names (SANs) containing the API server DNS and IP addresses or a wildcard certificate.
27+
** You can provide additional DNS names for each certificate.
28+
29+
. After the {microshift-short} service restarts, you must copy the generated `kubeconfig` files to the client.
30+
31+
. Configure additional CAs on the client system. For example, you can update CA bundles in the {op-system-base-full} truststore.
32+
33+
. The certificates and keys are read from the specified file location on the host. Testing and validation of configuration is done from the client.
34+
35+
. External server certificates are not automatically renewed. You must manually rotate your external certificates.
36+
37+
[NOTE]
38+
====
39+
If any validation fails, the {microshift-short} service skips the custom configuration and uses the default certificate to start. The priority is to continue the service uninterrupted. {microshift-short} logs errors when the service starts. Common errors include expired certificates, missing files, or incorrect IP addresses.
40+
====
41+
42+
[IMPORTANT]
43+
====
44+
Custom server certificates have to be validated against CA data configured in the trust root of the host operating system.
45+
====

‎modules/microshift-custom-ca-proc.adoc

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * microshift_security_compliance/microshift-custom-ca.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-custom-cas-configuring_{context}"]
7+
= Configuring custom certificate authorities
8+
9+
To configure externally generated certificates and domain names using custom certificate authorities (CAs), add them to the {microshift-short} `/etc/microshift/config.yaml` configuration file. You must also configure the host operating system trust root.
10+
11+
[NOTE]
12+
====
13+
Externally generated `kubeconfig` files are created in the `/var/lib/microshift/resources/kubeadmin/<hostname>/kubeconfig` directory. If you need to use `localhost` in addition to externally generated configurations, retain the original `kubeconfig` file in its default location. The `localhost` `kubeconfig` file uses the self-signed certificate authority.
14+
====
15+
16+
.Prerequisites
17+
* The OpenShift CLI (`oc`) is installed.
18+
* You have access to the cluster as a user with the cluster administration role.
19+
* The certificate authority has issued the custom certificates.
20+
* A {microshift-short} `/etc/microshift/config.yaml` configuration file exists.
21+
22+
.Procedure
23+
24+
. Copy the custom certificates you want to add to the trust root of the {microshift-short} host. Ensure that the
25+
certificate and private keys are only accessible to {microshift-short}.
26+
27+
. For each custom CA that you need, add an `apiServer` section called `namedCertificates` to the `/etc/microshift/config.yaml` {microshift-short} configuration file by using the following example:
28+
+
29+
[source,yaml]
30+
----
31+
apiServer:
32+
namedCertificates:
33+
- certPath: ~/certs/api_fqdn_1.crt <1>
34+
keyPath: ~/certs/api_fqdn_1.key <2>
35+
- certPath: ~/certs/api_fqdn_2.crt
36+
keyPath: ~/certs/api_fqdn_2.key
37+
names: <3>
38+
- api_fqdn_1
39+
- *.apps.external.com
40+
----
41+
<1> Add the full path to the certificate.
42+
<2> Add the full path to the certificate key.
43+
<3> Optional. Add a list of explicit DNS names. Leading wildcards are allowed. If no names are provided, the implicit names are extracted from the certificates.
44+
45+
. Restart the {microshift-service} to apply the certificates by running the following command:
46+
+
47+
[source,terminal]
48+
----
49+
$ systemctl microshift restart
50+
----
51+
52+
. Wait a few minutes for the system to restart and apply the custom server. New `kubeconfig` files are generated in the `/var/lib/microshift/resources/kubeadmin/` directory.
53+
54+
. Copy the `kubeconfig` files to the client. If you specified wildcards for the IP address, update the `kubeconfig` to remove the public IP address of the server and replace that IP address with the specific wildcard range you want to use.
55+
56+
. From the client, use the following steps:
57+
58+
.. Specify the `kubeconfig` to use by running the following command:
59+
+
60+
[source,terminal]
61+
----
62+
$ export KUBECONFIG=~/custom-kubeconfigs/kubeconfig <1>
63+
----
64+
<1> Use the location of the copied `kubeconfig` file as the path.
65+
66+
.. Check that the certificates are applied by using the following command:
67+
+
68+
[source,terminal]
69+
----
70+
$ oc --certificate-authority ~/certs/ca.ca get node
71+
----
72+
+
73+
.Example output
74+
[source,terminal]
75+
----
76+
oc get node
77+
NAME STATUS ROLES AGE VERSION
78+
dhcp-1-235-195.arm.example.com Ready control-plane,master,worker 76m v1.29.2
79+
----
80+
81+
.. Add the new CA file to the $KUBECONFIG environment variable by running the following command:
82+
+
83+
[source,terminal]
84+
----
85+
$ oc config set clusters.microshift.certificate-authority /tmp/certificate-authority-data-new.crt
86+
----
87+
88+
.. Verify that the new `kubeconfig` file contains the new CA by running the following command:
89+
+
90+
[source,terminal]
91+
----
92+
$ oc config view --flatten
93+
----
94+
+
95+
.Example externally generated `kubeconfig` file
96+
+
97+
[source,yaml]
98+
----
99+
apiVersion: v1
100+
clusters:
101+
- cluster:
102+
certificate-authority: /tmp/certificate-authority-data-new.crt <1>
103+
server: https://api.ci-ln-k0gim2b-76ef8.aws-2.ci.openshift.org:6443
104+
name: ci-ln-k0gim2b-76ef8
105+
contexts:
106+
- context:
107+
cluster: ci-ln-k0gim2b-76ef8
108+
user:
109+
name:
110+
current-context:
111+
kind: Config
112+
preferences: {}
113+
----
114+
<1> The `certificate-authority-data` section is not present in externally generated `kubeconfig` files. It is added with the `oc config set` command used previously.
115+
116+
.. Verify the `subject` and `issuer` of your customized API server certificate authority by running the following command:
117+
+
118+
[source,terminal]
119+
----
120+
$ curl --cacert /tmp/caCert.pem https://${fqdn_name}:6443/healthz -v
121+
----
122+
+
123+
.Example output
124+
----
125+
Server certificate:
126+
subject: CN=kas-test-cert_server
127+
start date: Mar 12 11:39:46 2024 GMT
128+
expire date: Mar 12 11:39:46 2025 GMT
129+
subjectAltName: host "dhcp-1-235-3.arm.eng.rdu2.redhat.com" matched cert's "dhcp-1-235-3.arm.eng.rdu2.redhat.com"
130+
issuer: CN=kas-test-cert_ca
131+
SSL certificate verify ok.
132+
----
133+
+
134+
[IMPORTANT]
135+
====
136+
Either replace the `certificate-authority-data` in the generated `kubeconfig` file with the new `rootCA` or add the `certificate-authority-data` to the trust root of the operating system. Do not use both methods.
137+
====
138+
139+
.. Configure additional CAs in the trust root of the operating system. For example, in the RHEL Client truststore on the client system. See link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/securing_networks/using-shared-system-certificates_securing-networks#the-system-wide-trust-store_using-shared-system-certificates[The system-wide truststore] for details.
140+
** Updating the certificate bundle with the configuration that contains the CA is recommended.
141+
** If you do not want to configure your certificate bundles, you can alternately use the `oc login localhost:8443 --certificate-authority=/path/to/cert.crt` command, but this method is not preferred.

‎modules/microshift-custom-ca-reserved-names.adoc

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * microshift_security_compliance/microshift-custom-ca.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="microshift-custom-ca-reserved-name-values_{context}"]
7+
= Custom certificates reserved name values
8+
9+
The following certificate problems cause {microshift-short} to ignore certificates dynamically and log an error:
10+
11+
* The certificate files do not exist on the disk or are not readable.
12+
* The certificate is not parsable.
13+
* The certificate overrides the internal certificates IPAddress/DNSNames in a `SubjectAlternativeNames` (SAN) field. Do not use a reserved name when configuring SANs.
14+
15+
.Reserved Names values
16+
[cols="<,<,<",options="header",]
17+
|===
18+
|Address |Type |Comment
19+
|`localhost` |DNS |
20+
|`127.0.0.1` |IP Address |
21+
|`10.42.0.0` |IP Address |Cluster Network
22+
|`10.43.0.0/16,10.44.0.0/16` |IP Address |Service Network
23+
|169.254.169.2/29 |IP Address |br-ex Network
24+
|`kubernetes.default.svc` |DNS |
25+
|`openshift.default.svc` |DNS |
26+
|`svc.cluster.local` |DNS |
27+
|===

‎modules/microshift-custom-ca-troubleshooting.adoc

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * microshift_security_compliance/microshift-custom-ca.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="microshift-custom-ca-troubleshootin_{context}"]
7+
= Troubleshooting custom certificates
8+
9+
To troubleshoot the implementation of custom certificates, you can take the following steps.
10+
11+
.Procedure
12+
13+
. From {microshift-short}, ensure that the certificate is served by the `kube-apiserver` and verify that the certificate path is appended to the `--tls-sni-cert-key` FLAG by running the following command:
14+
+
15+
[source,terminal]
16+
----
17+
$ journalctl -u microshift -b0 | grep tls-sni-cert-key
18+
----
19+
+
20+
.Example output
21+
[source,terminal]
22+
----
23+
Jan 24 14:53:00 localhost.localdomain microshift[45313]: kube-apiserver I0124 14:53:00.649099 45313 flags.go:64] FLAG: --tls-sni-cert-key="[/home/eslutsky/dev/certs/server.crt,/home/eslutsky/dev/certs/server.key;/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-external-signer/kube-external-serving/server.key;/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-localhost-signer/kube-apiserver-localhost-serving/server.key;/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.crt,/var/lib/microshift/certs/kube-apiserver-service-network-signer/kube-apiserver-service-network-serving/server.key
24+
----
25+
26+
. From the client, ensure that the `kube-apiserver` is serving the correct certificate by running the following command:
27+
+
28+
[source,terminal]
29+
----
30+
$ openssl s_client -connect <SNI_ADDRESS>:6443 -showcerts | openssl x509 -text -noout -in - | grep -C 1 "Alternative\|CN"
31+
----

0 commit comments

Comments
 (0)