diff --git a/modules/gitops-configure-hashicorp-vault-as-a-secrets-provider-on-openshift-with-gitops.adoc b/modules/gitops-configure-hashicorp-vault-as-a-secrets-provider-on-openshift-with-gitops.adoc new file mode 100644 index 000000000000..184e2d770b77 --- /dev/null +++ b/modules/gitops-configure-hashicorp-vault-as-a-secrets-provider-on-openshift-with-gitops.adoc @@ -0,0 +1,34 @@ +// Module is included in the following assemblies: +// +// * securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc + +:_mod-docs-content-type: CONCEPT +[id="gitops-configure-hashicorp-vault-as-a-secrets-provider-on-openshift-with-gitops_{context}"] += Configure HashiCorp Vault as a secrets provider on OpenShift with GitOps + +You can configure HashiCorp Vault as a secrets provider by using the Secrets Store CSI Driver Operator on the {OCP}. When combined with GitOps workflows managed by Argo CD, this setup enables you to securely retrieve secrets from Vault and inject it into your applications running on OpenShift. + +Structure the {gitops-shortname} repository and configure the Vault CSI provider to integrate with the Secrets Store CSI Driver in {OCP}. + +The following sample {gitops-shortname} repository layout is used for integrating Vault with your application. + +.Example directory structure in {gitops-shortname} repository +---- +├── config +│   ├── argocd # <1> +│   │   ├── vault-secret-provider-app.yaml +│   │   ├── ... +│── environments +│   ├── dev +│   │   ├── apps +│   │   │   ├── demo-app +│   │   │   ├── manifest # <2> +│   │   │      | ├── secretProviderClass.yaml +│   │   │      | ├── serviceAccount.yaml +│   │   │      | ├── deployment.yaml +│   │   │   ├── argocd # <3> +│   │   │      ├── demo-app.yaml +---- +<1> `config/argocd/` - Stores Argo CD Application definitions for cluster-wide tools like the Vault CSI provider. +<2> `environments//apps//manifest/`: Contains Kubernetes manifests specific to an application in a particular environment. +<3> `environments//apps//argocd/`: Contains the Argo CD Application definition that deploys the application and its resources. \ No newline at end of file diff --git a/modules/gitops-configuring-gitops-managed-resources-to-use-vault-mounted-secrets.adoc b/modules/gitops-configuring-gitops-managed-resources-to-use-vault-mounted-secrets.adoc new file mode 100644 index 000000000000..c3b5deb9aecd --- /dev/null +++ b/modules/gitops-configuring-gitops-managed-resources-to-use-vault-mounted-secrets.adoc @@ -0,0 +1,129 @@ +// Module is included in the following assemblies: +// +// * securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc + +:_mod-docs-content-type: PROCEDURE +[id="gitops-configuring-gitops-managed-resources-to-use-vault-mounted-secrets_{context}"] += Configuring GitOps managed resources to use Vault-mounted secrets + +Securely inject secrets from HashiCorp Vault into GitOps-managed Kubernetes workloads using the Secrets Store CSI driver and Vault provider. The secrets are mounted as files in the pod's filesystem, allowing applications to access the data without storing it in Kubernetes Secret objects. + +.Procedure + +. Creating the `SecretProviderClass`. + +.. Create a `SecretProviderClass` resource in the application's manifest directory for example, `environments/dev/apps/demo-app/manifest/secretProviderClass.yaml`. This resource defines how the Secrets Store CSI driver retrieves secrets from Vault. ++ +.Example `vault-secret-provider-app.yaml` file +[source,yaml] +---- +apiVersion: secrets-store.csi.x-k8s.io/v1 + kind: SecretProviderClass + metadata: + name: demo-app-creds + namespace: demo-app + spec: + provider: vault #<1> + parameters: + vaultAddress: http://vault.vault-csi-provider:8200 # .:port #<2> + roleName: app #<3> + objects: | #<4> + - objectName: "demoAppUsername" + secretPath: "secret/demo/config" + secretKey: "username" + - objectName: "demoAppPassword" + secretPath: "secret/demo/config" + secretKey: "password" +---- +<1> `` - Specifies the name of the HashiCorp Vault. +<2> `` - Specifies the network address of the Vault server. Adjust this based on your Vault setup, such as, in-cluster service or an external URL. +<3> `` - Specifies the Vault Kubernetes authentication role used by the application Service Account. +Describes an array that defines which secrets to retrieve and how to map them to file names. +<4> `` - Specifies an array that defines which secrets to retrieve and how to map them to file names. The `secretPath` for KV v2 includes `/data/`. + +. Create an Application, such as, `ServiceAccount`. + +.. Create a Kubernetes `ServiceAccount` for the application workload. The `ServiceAccount` name must match the `bound_service_account_names` value defined in the Vault Kubernetes authentication role. Store the manifest in the GitOps repository, for example, `environments/dev/apps/demo-app/manifest/serviceAccount.yaml`. ++ +.Example `ServiceAccount.yaml` file +[source,yaml] +---- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: demo-app-sa + namespace: demo-app +---- + +. Create the Application deployment: + +.. Modify the application's deployment to use the designated `ServiceAccount` and mount secrets using the CSI volume. Store the updated manifest in the GitOps repository, for example, `environments/dev/apps/demo-app/manifest/deployment.yaml`: ++ +.Example `deployment.yaml` file +[source,yaml] +---- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: app + namespace: demo-app + labels: + app: demo +spec: + replicas: 1 + selector: + matchLabels: + app: demo + template: + metadata: + labels: + app: demo + spec: + serviceAccountName: demo-app-sa # <1> + containers: + - name: app + image: nginxinc/nginx-unprivileged:latest + volumeMounts: # <2> + - name: vault-secrets + mountPath: /mnt/secrets-store + readOnly: true + volumes: # <3> + - name: vault-secrets + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: demo-app-creds +---- +<1> `serviceAccountName` - Assigns the Kubernetes `ServiceAccount` name, for example, `demo-app-sa`, used by the application pod. This `ServiceAccount` is fundamental for authenticating with HashiCorp Vault, as it is linked to a Vault role that grants permissions to retrieve the necessary secrets. +<2> `volumeMounts` - Mounts the vault-secrets volume into the container at the `/mnt/secrets-store` directory. +<3> `volumes` - Defines the vault-secrets volume using the `secrets-store.csi.k8s.io` driver and references the `demo-app-creds` `SecretProviderClass`. + +. Define the Argo CD application for the workload: + +.. Define an Argo CD application resource to deploy application components such as `ServiceAccount`, `SecretProviderClass`, and `Deployment` from the GitOps repository. Store the Argo CD manifest in a directory location, such as, `environments/dev/apps/demo-app/argocd/demo-app.yaml`. ++ +.Example `demo-app.yaml` file +[source,yaml] +---- +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: demo-app + namespace: openshift-gitops +spec: + project: default + source: + repoURL: https://your-git-repo-url.git + targetRevision: HEAD + path: environments/dev/apps/demo-app/manifest + destination: + server: https://kubernetes.default.svc + namespace: demo-app + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +---- \ No newline at end of file diff --git a/modules/gitops-installing-and-configuring-vault-to-store-a-secret.adoc b/modules/gitops-installing-and-configuring-vault-to-store-a-secret.adoc new file mode 100644 index 000000000000..96681df1e751 --- /dev/null +++ b/modules/gitops-installing-and-configuring-vault-to-store-a-secret.adoc @@ -0,0 +1,105 @@ +// Module is included in the following assemblies: +// +// * securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc + +:_mod-docs-content-type: PROCEDURE +[id="gitops-installing-and-configuring-vault-to-store-a-secret_{context}"] += Installing the Vault CSI Provider using {gitops-shortname} + +After deploying Vault using Argo CD and applying the necessary SCC permissions and DaemonSet patches, initialize Vault, unseal it, and configure Kubernetes authentication to enable secure secret storage and access. + +.Procedure + +. Access the Vault Pod. + +.. If Vault is running within your {OCP} cluster, for example, as the `vault-0` pod in the `vault-csi-provider` namespace, run the following command to access the Vault CLI inside the pod: ++ +[source,terminal] +---- +$ oc exec -it vault-0 -n vault-csi-provider -- /bin/sh +---- + +. Initialize Vault. + +.. If your Vault instance is not yet initialized, run the following command: ++ +[source,terminal] +---- +$ vault operator init +---- ++ +As a result, the following output is displayed. ++ +[source,output] +---- +5 Unseal Keys - required to unseal the Vault. +Initial Root Token - required to log in and configure Vault. +---- ++ +[IMPORTANT] +==== +Store these credentials securely. At least 3 out of 5 unseal keys are required to unseal Vault. If the keys are lost, access to stored secrets is permanently blocked. +==== + +. Unseal Vault. + +.. Vault starts in a sealed state. Run the following commands to use three of the five Unseal Keys obtained in the earlier step: ++ +[source,terminal] +---- +$ vault operator unseal + vault operator unseal + vault operator unseal +---- ++ +Once unsealed, the Vault becomes active and ready for use. + +. Log into Vault. + +.. To use the root token to log in to Vault, run the following command: ++ +[source,terminal] +---- +$ vault login +---- ++ +This provides administrator access to enable and configure secret engines and authentication methods. + +. Enable Kubernetes Authentication in Vault. + +.. Run the following command to enable Kubernetes authentication in Vault. ++ +[source,terminal] +---- +$ vault auth enable kubernetes +---- ++ +This allows Kubernetes workloads, for example, pods, to authenticate with Vault using their service accounts. + +. Configure Kubernetes authentication method in Vault. + +.. To configure Vault for communicating with the Kubernetes API, run the following command: ++ +[source,terminal] +---- +$ vault write auth/kubernetes/config \ +issuer="https://kubernetes.default.svc" \ +token_reviewer_jwt="$(cat/var/run/secrets/kubernetes.io/serviceaccount/token)" \ +kubernetes_host="https://${KUBERNETES_PORT_443_TCP_ADDR}:443" \ +kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt +---- ++ +As a result, the following output is displayed. ++ +[source,output] +---- +Success! Data written to: auth/kubernetes/config +---- ++ +Where: ++ +* `` is the name of the Kubernetes token issuer URL. +* `` is a JSON Web Token (JWT) that Vault uses to call the Kubernetes `TokenReview` API and to validate service account tokens. +* `` is the URL that Vault uses to communicate with the Kubernetes API server. +* `` is the CA certificate that Vault uses for secure communication with the Kubernetes API server. + diff --git a/modules/gitops-installing-the-vault-csi-provider-using-gitops.adoc b/modules/gitops-installing-the-vault-csi-provider-using-gitops.adoc new file mode 100644 index 000000000000..a9a6479972c1 --- /dev/null +++ b/modules/gitops-installing-the-vault-csi-provider-using-gitops.adoc @@ -0,0 +1,115 @@ +// Module is included in the following assemblies: +// +// * securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc + +:_mod-docs-content-type: PROCEDURE +[id="gitops-installing-the-vault-csi-provider-using-gitops_{context}"] += Installing the Vault CSI Provider using {gitops-shortname} + +Install the Vault CSI provider by deploying an Argo CD Application that uses HashiCorp's official Helm chart. This method follows {gitops-shortname} best practices by managing the installation declaratively through a version-controlled Argo CD Application resource. + +.Prerequisites + +* You are logged in to the OpenShift Container Platform cluster as an administrator. +* You have access to the {OCP} web console. +* The link:https://docs.openshift.com/container-platform/latest/nodes/pods/nodes-pods-secrets-store.html#persistent-storage-csi-secrets-store-driver-install_nodes-pods-secrets-store[SSCSI Driver Operator] is installed on your cluster. +* You installed xref:../installing_gitops/installing-openshift-gitops.adoc#installing-openshift-gitops[{gitops-title}] on your {OCP} cluster. +* You have a {gitops-shortname} repository ready to use the secrets. + +.Procedure + +. Creating the Argo CD Application resource for the Vault CSI Provider. + +.. Create an Argo CD Application resource to deploy the Vault CSI provider. Add this resource to your {gitops-shortname} repository, for example, `config/argocd/vault-secret-provider-app.yaml`: ++ +.Example `vault-secret-provider-app.yaml` file +[source,yaml] +---- +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: vault-secret-provider-app + namespace: openshift-gitops +spec: + destination: + namespace: vault-csi-provider + server: https://kubernetes.default.svc + project: default + source: + repoURL: https://helm.releases.hashicorp.com + chart: vault + targetRevision: 0.30.0 + helm: + releaseName: vault + values: | + csi: + enabled: true + server: + enabled: true + dataStorage: + enabled: false + injector: + enabled: false + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true + ignoreDifferences: + - kind: DaemonSet + group: apps + jsonPointers: + - /spec/template/spec/containers/0/securityContext/privileged + +---- ++ +[NOTE] +==== +The `server.enabled: true` and `dataStorage.enabled: false` settings in the Helm values deploy a HashiCorp Vault server instance using ephemeral storage. This setup is suitable for development or testing environments. For production, you can enable `dataStorage` with a persistent volume (PV) or use an external Vault cluster and set `server.enabled` to `false`. If a Vault server is already deployed, you can set `server.enabled` to `false`. +==== + +. Apply the `vault-secret-provider-app.yaml` file from the {gitops-shortname} repository to your cluster: ++ +[source,terminal] +---- +$ oc apply -f vault-secret-provider-app.yaml +---- ++ +After deploying the Vault CSI provider, the `vault-csi-provider` DaemonSet may fail to run. This issue occurs because {OCP} restricts privileged containers by default. In addition, the Vault CSI provider and the Secrets Store CSI Driver require access to `hostPath` mounts, which {OCP} blocks unless the pods run as privileged. + +.. To resolve permission issues in {OCP}: + +... Patch the `vault-csi-provider` DaemonSet to run its containers as privileged: ++ +[source,terminal] +---- +$ oc patch daemonset vault-csi-provider -n vault-csi-provider --type=json --patch='[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value":{"privileged":true}}] +---- + +... Grant the Secrets Store CSI Driver service account access to the privileged Security Context Constraints (SCC) in {OCP}. ++ +[source,terminal] +---- +$ oc adm policy add-scc-to-user privileged \ system:serviceaccount:openshift-cluster-csi-drivers:secrets-store-csi-driver-operator +---- + +... Grant the Vault CSI Provider service account access to the privileged Security Context Constraints (SCC) in {OCP}. ++ +[source,terminal] +---- +$ oc adm policy add-scc-to-user privileged \ +system:serviceaccount:vault-csi-provider:vault-csi-provider +---- ++ +[NOTE] +==== +If `server.enabled` is set to `true` in the Helm chart, the Vault server pods run with specific user IDs (UIDs) or group IDs (GIDs) that {OCP} blocks by default. +==== + +... Grant the Vault server service account the required Security Context Constraints (SCC) permissions. ++ +[source,terminal] +---- +$ oc adm policy add-scc-to-user anyuid system:serviceaccount:vault-csi-provider:vault +---- \ No newline at end of file diff --git a/modules/gitops-managing-secrets-policies-and-roles-in-vault.adoc b/modules/gitops-managing-secrets-policies-and-roles-in-vault.adoc new file mode 100644 index 000000000000..952efe41e03e --- /dev/null +++ b/modules/gitops-managing-secrets-policies-and-roles-in-vault.adoc @@ -0,0 +1,62 @@ +// Module is included in the following assemblies: +// +// * securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc + +:_mod-docs-content-type: PROCEDURE +[id="gitops-managing-secrets-policies-and-roles-in-vault_{context}"] += Managing Secrets, Policies, and Roles in Vault + +To create a secret in Vault, define a Vault policy and configure a Kubernetes authentication role that enables a Kubernetes workload to retrieve the secret securely. + +.Procedure + +. Enable the KV Secrets Engine + +.. Use the Key-Value (KV) Version 2 secrets engine to store arbitrary secrets with versioning support. Run the following command to enable the KV secrets engine at the path secret/: ++ +[source,terminal] +---- +$ vault secrets enable -path=secret/ kv +---- + +. Store a secret in Vault. + +.. Store a secret using the KV Version 2 secrets engine. Run the following command to store the secret data, username and password, at path `secret/demo/config`: ++ +[source,terminal] +---- +$ vault kv put secret/demo/config username="demo-user" password="demo-pass" +---- + +. Create a Vault policy. + +.. To create a policy that grants read access to the secret, run the following command: ++ +[source,terminal] +---- +$ vault policy write demo-app-policy -<` is the name of the Kubernetes service account that Vault trusts. +* `` is the name of the namespace where the service account is located. +* `` is the name of the attached Vault policy. +* `` is the `Time-to-live` value issued for the token. diff --git a/modules/gitops-managing-secrets-using-sscsid-with-gitops-overview.adoc b/modules/gitops-managing-secrets-using-sscsid-with-gitops-overview.adoc index bdd5fb83a7e5..8fdec4de0a4d 100644 --- a/modules/gitops-managing-secrets-using-sscsid-with-gitops-overview.adoc +++ b/modules/gitops-managing-secrets-using-sscsid-with-gitops-overview.adoc @@ -34,6 +34,7 @@ The following secrets store providers are available for use with the Secrets Sto * AWS Secrets Manager * AWS Systems Manager Parameter Store * Microsoft Azure Key Vault +* HashiCorp Vault As an example, consider that you are using AWS Secrets Manager as your secrets store provider with the SSCSI Driver Operator. The following example shows the directory structure in {gitops-shortname} repository that is ready to use the secrets from AWS Secrets Manager: diff --git a/modules/gitops-verifying-secret-injection.adoc b/modules/gitops-verifying-secret-injection.adoc new file mode 100644 index 000000000000..10d417228e1b --- /dev/null +++ b/modules/gitops-verifying-secret-injection.adoc @@ -0,0 +1,42 @@ +// Module is included in the following assemblies: +// +// * securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc + +:_mod-docs-content-type: PROCEDURE +[id="gitops-verifying-secret-injection_{context}"] += Verifying secret injection + +Verify the secret injection to ensure that Vault contains the expected values. + +.Procedure + +. Check the Pod status. + +.. After the Argo CD Application has synced and all the resources are deployed, verify that the application pod is running successfully in the `demo-app` namespace. Run the following command: ++ +[source,terminal] +---- +$ oc get pods -n demo-app +---- + +. Open the Shell session. + +.. Use the name of the application pod to open a shell session. Replace `` with the actual pod name. ++ +[source,terminal] +---- +$ oc exec -it -n demo-app -- sh +---- + +. Verify mounted secrets. + +.. To verify that the secrets are mounted at the expected path, run the following command: ++ +[source,terminal] +---- +$ ls -l /mnt/secrets-store + cat /mnt/secrets-store/demoAppUsername + cat /mnt/secrets-store/demoAppPassword +---- ++ +Verify that the mounted secret files `demoAppUsername` and `demoAppPassword` contain the expected values from Vault. \ No newline at end of file diff --git a/securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc b/securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc index b5f61c3df86f..4930dfd1129d 100644 --- a/securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc +++ b/securing_openshift_gitops/managing-secrets-securely-using-sscsid-with-gitops.adoc @@ -28,10 +28,28 @@ include::modules/gitops-managing-secrets-using-sscsid-with-gitops-overview.adoc[ include::modules/gitops-storing-aws-secret-manager-resources-in-gitops-repository.adoc[leveloffset=+1] // Configuring SSCSI driver to mount secrets from AWS Secrets Manager -include::modules/gitops-configuring-sscsi-driver-to-mount-secrets-from-aws-secrets-manager.adoc[leveloffset=+1] +include::modules/gitops-configuring-sscsi-driver-to-mount-secrets-from-aws-secrets-manager.adoc[leveloffset=+2] // Configuring GitOps managed resources to use mounted secrets -include::modules/gitops-configuring-gitops-managed-resources-to-use-mounted-secrets.adoc[leveloffset=+1] +include::modules/gitops-configuring-gitops-managed-resources-to-use-mounted-secrets.adoc[leveloffset=+2] + +// Configure HashiCorp Vault as a secrets provider on OpenShift with GitOps +include::modules/gitops-configure-hashicorp-vault-as-a-secrets-provider-on-openshift-with-gitops.adoc[leveloffset=+1] + +// Installing the Vault CSI Provider using GitOps +include::modules/gitops-installing-the-vault-csi-provider-using-gitops.adoc[leveloffset=+2] + +// Installing and configuring Vault to store a Secret +include::modules/gitops-installing-and-configuring-vault-to-store-a-secret.adoc[leveloffset=+2] + +// Managing Secrets, Policies, and Roles in Vault +include::modules/gitops-managing-secrets-policies-and-roles-in-vault.adoc[leveloffset=+2] + +// Configuring GitOps managed resources to use Vault-mounted secrets +include::modules/gitops-configuring-gitops-managed-resources-to-use-vault-mounted-secrets.adoc[leveloffset=+2] + +// Verifying secret injection +include::modules/gitops-verifying-secret-injection.adoc[leveloffset=+2] [role="_additional-resources"] [id="additional-resources_{context}"]