External secrets

.vscode/settings.json

@@ -0,0 +1,29 @@
+{
+    "cSpell.words": [
+        "alertmanager",
+        "azurekv",
+        "cicd",
+        "creds",
+        "crunchydb",
+        "Entra",
+        "externalsecret",
+        "golddr",
+        "IDIR",
+        "imagestreams",
+        "Kamloops",
+        "KLAB",
+        "Kyverno",
+        "licenseplate",
+        "linenums",
+        "myapp",
+        "nonprod",
+        "opensource",
+        "Patroni",
+        "pids",
+        "pymdownx",
+        "rolebinding",
+        "serviceaccount",
+        "Sysdig",
+        "techdocs"
+    ]
+}

mkdocs.yml

@@ -80,6 +80,7 @@ nav:
 - Secrets management:
   - Vault getting started guide: docs/secrets-management/vault-getting-started-guide.md
   - Vault secrets management: docs/secrets-management/vault-secrets-management-service.md
+  - External secrets: docs/secrets-management/external-secrets.md
 - Training and learning:
   - Training from the Platform Services Team: docs/training-and-learning/training-from-the-platform-services-team.md
   - External training resources: docs/training-and-learning/training-external-resources.md

src/docs/secrets-management/example_secretstore_azure_key_vault.md

@@ -0,0 +1,139 @@
+# Example SecretStore - Azure Key Vault
+
+## Summary
+
+In order to use Azure Key Vault with the [External Secrets Operator](external-secrets.md), you need to create a Service Principal with the right permissions. You then store the Service Principal’s credentials in a Kubernetes Secret in each namespace where you’ll create `SecretStore` and `ExternalSecret` resources.
+
+## Requirements
+
+To complete this setup, you need:
+
+* Access to your Azure Key Vault
+* Access to your OpenShift namespaces
+* Docker or Podman, if you're using the Azure CLI
+
+You can create the Service Principal using either the Azure CLI or the Azure Portal. This guide uses the CLI method. If you prefer the portal, see [Register a Microsoft Entra app and create a service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal).
+
+## Official documentation
+
+[External Secrets Operator - Azure Key Vault Provider](https://external-secrets.io/latest/provider/azure-key-vault/)
+
+## Start the Azure CLI Container
+
+We recommend running the Azure CLI in a Docker or Podman container. Installing the CLI directly on your machine requires many dependencies, which might conflict with other tools or take up unnecessary space if you only need it for this task.
+
+Make sure you have a running Docker or Podman environment before starting.
+
+Start the container:
+
+```
+podman run -it mcr.microsoft.com/azure-cli:cbl-mariner2.0
+```
+
+Once it's running and you have a command prompt, run `az` to see version and help information.
+
+```
+az
+```
+
+Log in to Azure by running `az login` in the container and following the instructions.
+
+```
+root [ / ]# az login
+To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ... to authenticate.
+```
+
+Make a note of the subscription ID in the output of the login command.
+
+## Create the Service Principal
+
+To create the Service Principal, make sure you have the following information:
+
+* Your Azure subscription ID
+* The name of your Key Vault
+* The resource group of your Key Vault
+
+You’ll use this information when you run the Azure CLI commands in the next steps:
+```
+export SUBSCRIPTION_ID="your-subscription-ID"
+export KEY_VAULT_NAME="your-key-vault-name"
+export RESOURCE_GROUP="your-key-vaults-resource-group"
+export SP_NAME="name-of-service-principal-to-create"
+```
+
+Run the command to create the service principal:
+```
+az ad sp create-for-rbac --name "${SP_NAME}" --role "Key Vault Secrets User" --scopes /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${KEY_VAULT_NAME} --sdk-auth
+```
+
+Save a copy of the output from this command - you'll need `clientId`, `clientSecret`, and `tenantId`.  Set the client credentials as environment variables if you'd like to copy the command below to create the Secret.
+```
+export CLIENT_ID=clientId_from_output
+export CLIENT_SECRET=clientSecret_from_output
+```
+
+## Create the OpenShift Secret
+
+First, create a Secret in your OpenShift namespace to store your Azure Service Principal credentials.  You can use the UI if you like, or use the following command:
+```
+oc create secret generic azure-key-vault-creds --from-literal=clientId=${CLIENT_ID} --from-literal=clientSecret=${CLIENT_SECRET}
+```
+
+## Assign permissions to the Service Principal
+
+Using the Azure CLI, get a list of Service Principals:
+
+```
+az ad sp list --show-mine
+```
+
+If you see more than one, look for the one with the `displayName` that is equal to your new Service Principal.  Find its `id`  for that entry and assign it to an environment variable.
+```
+export OBJECT_ID="the-service-principals-id"
+```
+
+Or, use `jq` to extract it automatically:
+```
+export OBJECT_ID=`az ad sp list --show-mine | jq -r ".[] | select(.displayName == \"${SP_NAME}\") | .id"`
+```
+
+Now assign the right permissions to your Service Principal so it can access Secrets in Azure Key Vault:
+```
+az keyvault set-policy --name ${KEY_VAULT_NAME} --object-id ${OBJECT_ID} --secret-permissions get list
+```
+
+## Create a SecretStore
+Next, create a YAML manifest for the `SecretStore`.  Be sure to enter the correct values for the `tenantId` and the name of the Secret that you created above.
+```
+apiVersion: external-secrets.io/v1beta1
+kind: SecretStore
+metadata:
+  name: azure-key-vault
+  namespace: abc123-dev
+spec:
+  provider:
+    azurekv:
+      tenantId: "MY_TENANT_ID"
+      vaultUrl: https://my-key-vault-name.vault.azure.net/
+      authSecretRef:
+        clientId:
+          name: azure-key-vault-creds
+          key: clientId
+        clientSecret:
+          name: azure-key-vault-creds
+          key: clientSecret
+```
+
+After applying the YAML manifest, check the status of the new SecretStore.  It should show as ready.
+```
+status:
+  capabilities: ReadWrite
+  conditions:
+    - lastTransitionTime: "2025-05-21T17:43:07Z"
+      message: store validated
+      reason: Valid
+      status: "True"
+      type: Ready
+```
+
+Once the SecretStore is ready, you can create an [ExternalSecret](external-secrets.md#create-an-externalsecret) to sync your secrets.

src/docs/secrets-management/external-secrets.md

@@ -0,0 +1,95 @@
+---
+title: External secrets
+
+slug: external-secrets-operator
+
+description: The External Secrets Operator can link your OpenShift namespace with an external secrets management service.
+
+keywords: 
+
+page_purpose: Describes the purpose and use of the External Secrets Operator
+
+audience: developer, technical lead
+
+author: Ian Watts
+
+content_owner: Ian Watts
+
+sort_order: 3
+---
+
+# External Secrets
+
+The **External Secrets Operator (ESO)** is a Kubernetes operator that connects to external secret management systems like [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault/) and [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html).  The operator reads information from external APIs (systems) and automatically adds them to Kubernetes as Secrets.
+
+ESO's main purpose to keep your Kubernetes Secrets in sync with external APIs. It uses custom resources `ExternalSecret` and `SecretStore` to provide a user-friendly abstraction for the external API that stores and manages the lifecycle of the secrets for you.
+
+ESO is available in each cluster. It's **self-serve**, so you can:
+
+* Choose from many different secret management services
+* Use the same service across your hybrid cloud environment
+* Add redundancy to your secrets management set up
+
+For more details, visit the [official External Secrets Operator documentation](https://external-secrets.io/latest/).
+
+Note that OpenShift Secrets are now encrypted on disk, which resolves what was previously a security concern.
+
+## How External Secrets Operator works
+
+To connect to an external secrets management system, you create two resources:
+
+* A `SecretStore`, which sets up the connection and credentials
+* One or more `ExternalSecrets`, which define the specific secrets and keys to copy into OpenShift
+
+Create both resources  in each namespace where you want to replicate secrets.
+
+## Create a SecretStore
+The `SecretStore`  resource stores the address and credentials for your secrets management service.  The setup depends on which service you're using.
+
+Check the [provider list](https://external-secrets.io/latest/provider/aws-secrets-manager/) for setup instructions specific to your service.
+
+If you are using Azure Key Vault, see the [Example SecretStore - Azure Key Vault](example_secretstore_azure_key_vault.md).
+
+## Create an ExternalSecret
+An `ExternalSecret` connects to a `SecretStore` and lists the specific secrets you want to copy from the external service. Each `ExternalSecret` creates one OpenShift secret. 
+
+The key-value pairs or other secret data you define in the `ExternalSecret`  are automatically added to the OpenShift secret.
+
+Here is an example:
+```
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: my-app-1
+  namespace: abc123-dev
+spec:
+  secretStoreRef:
+    kind: SecretStore
+    # The name of your SecretStore
+    name: azure-key-vault
+  target:
+    # The name of the Secret in OpenShift.
+    # It will be created if it does not already exist.
+    name: my-app-1
+  data:
+    - remoteRef:
+        # The name of the key in the external secrets system
+        key: dev-db-user
+      # The name of the key in the OpenShift secret
+      secretKey: db-user
+    - remoteRef:
+        key: dev-db-pass
+      secretKey: db-pass
+```
+
+For more information when creating an ExternalSecret, use the `oc` CLI, such as:
+```
+oc explain externalsecret.spec
+```
+
+```
+oc explain externalsecret.spec.refreshInterval
+```
+
+or edit it in the YAML view in the OpenShift UI and click on the the 'View sidebar' link.
+

src/docs/secrets-management/vault-getting-started-guide.md

@@ -20,10 +20,10 @@ sort_order: 1
 
 # Vault getting started guide
 
-The Platform Services team has deployed Hashicorp's Vault application with disaster recovery in our Gold service clusters (Gold Kamloops and Gold Calgary), the service is available to all B.C. government development teams on **all clusters** including: Silver, Gold/Golddr and Emerald.
+The Platform Services team has deployed Hashicorp's Vault application with disaster recovery in our Gold service clusters (Gold Kamloops and Gold Calgary). The service is available to all B.C. government development teams on **all clusters** including: Silver, Gold/GoldDR and Emerald.
 
 The Vault service is available at [https://vault.developer.gov.bc.ca/](https://vault.developer.gov.bc.ca/).
-If you are running a test project in the lab clusters, such as KLAB, use a separate Vault service available at [https://vault-lab.developer.gov.bc.ca/](https://vault-lab.developer.gov.bc.ca/).  The lab Vault instance is meant to be used for testing purposes by the Platform Services team, there could be frequent changes and outages with the service!
+If you are running a test project in the lab clusters, such as KLAB, use the lab instance of Vault at [https://vault-lab.developer.gov.bc.ca/](https://vault-lab.developer.gov.bc.ca/).
 
 In this guide, you will learn how to access Vault and deploy a working example using sample secrets from your Vault Mount Points. Mount Points can be thought of as "MiniVaults" or related to paths within a Linux filesystem that can be locked down and secured.
 
@@ -53,7 +53,7 @@ User Access gives authorized users the ability to create and write secrets, whil
 
 User Access is controlled through a combination of Red Hat Single-Sign On (KeyCloak) and automation integrated with the [Platform Services Registry](https://registry.developer.gov.bc.ca/).
 
-For each OpenShift Project Set ($LICENSE_PLATE-dev/test/prod/tools - eg: abc123-dev, abc123-test, abc123-prod, abc123-tools), up to two technical contacts would be grant access to Vault due to license limitation. These technical contacts will be given write access to the Mount Points set up by the registry.
+For each OpenShift Project Set ($LICENSE_PLATE-dev/test/prod/tools - eg: abc123-dev, abc123-test, abc123-prod, abc123-tools), just the two technical contacts are granted access, due to the user limits of the Vault license. These technical contacts are given write access to the Mount Points set up by the registry.
 
 #### Log in to Vault UI
 
@@ -288,11 +288,11 @@ vault.hashicorp.com/agent-limits-mem: '100Mi'
 
 Note the additional line under spec with `2. Service Account`. This is the service account that is needed to connect to the Vault. This account has been created already for you in OpenShift so on the surface it's straight forward.
 
-There's a issue to be aware of with using this service account (SA): your application container will also use this SA to pull images and for management. So if you are using an image from tools namespace, make sure to add an ImagePuller rolebinding to the SA. If you are using Artifactory, add the imagePullSecrets to either the SA or the deployment spec.
+There's an issue to be aware of with using this service account (SA): your application container will also use this SA to pull images and for management. So if you are using an image from your tools namespace, make sure to add an ImagePuller rolebinding to the SA. If you are using Artifactory, add the imagePullSecrets to either the SA or the deployment spec.
 
 **Part 3 - How to use secrets in application container**
 
-The Vault init container will create local files with the secret key-value pairs pulled from Vault server. They are saved at the path `/vault/secrets/<name>` where name is the secret name you specified in the annotation above. Unlike OpenShift secrets (which is not recommended as they are only encoded but not encrypted), they are not presented to app containers as environment variables directly. You will have to source the secret files first! See `3. how to use the secret` from the above example.
+The Vault init container will create local files with the secret key-value pairs pulled from Vault. They are saved at the path `/vault/secrets/<name>` where name is the secret name you specified in the annotation above. Unlike OpenShift secrets, they are not presented to app containers as environment variables directly. You will have to source the secret files first! See `3. how to use the secret` from the above example.
 
 > Note: there are different ways to use the secrets, check out [more examples here](https://www.vaultproject.io/docs/platform/k8s/injector/examples#vault-agent-injector-examples).
 

src/docs/secrets-management/vault-secrets-management-service.md

@@ -35,6 +35,7 @@ Non-secrets may be stored in Vault if desired.
 - [What does it cost?](#what-does-it-cost)
 - [Support roles, processes, communications (platform operations)](#support-roles-processes-communications-platform-operations)
 - [Service delivery](#service-delivery)
+- [Alternatives](#alternatives)
 
 ## Features and functions
 
@@ -78,7 +79,7 @@ The best source of help is the vibrant community of product teams using Vault fo
 
 You can find this highly talented and knowledgeable group in the [#devops-vault channel on Rocket.Chat](https://chat.developer.gov.bc.ca/channel/devops-vault).
 
-For help beyond this contact one of the Vault administrators via the [#devops-sos channel on Rocket.Chat](https://chat.developer.gov.bc.ca/channel/devops-sos).
+For help beyond this contact one of the Vault administrators via the [#devops-operations channel on Rocket.Chat](https://chat.developer.gov.bc.ca/channel/devops-operations).
 
 ## What does it cost?
 
@@ -92,7 +93,7 @@ Vault interfaces with Kubernetes services to provide authentication via service
 
 Rocket.Chat is the primary mode of communication. Specifically, the [#devops-vault](https://chat.developer.gov.bc.ca/channel/devops-vault) channel should be used to engage the community for best practices, configuration and troubleshooting questions.
 
-For cluster wide service notifications that may impact Vault monitor, use the [#devops-alerts channels in Rocket.Chat.](https://chat.developer.gov.bc.ca/channel/devops-alerts)
+For cluster wide service notifications that may impact Vault monitor, use the [#devops-alerts channel in Rocket.Chat.](https://chat.developer.gov.bc.ca/channel/devops-alerts)
 
 For teams without Rocket.Chat access or to escalate a question or concern, contact us by email at [PlatformServicesTeam@gov.bc.ca](mailto:PlatformServicesTeam@gov.bc.ca). 
 
@@ -107,18 +108,21 @@ As part of project onboarding, Kubernetes service accounts are generated for you
 Product teams can choose to use Vault or ETCD backed Kubernetes Secrets, but it is recommended that Vault be used for secrets.
 
 ### Change management
-Any changes to the Vault Secrets Management tool will be communicated via #devops-vault and #internal-devops-vault Rocket.Chat channels. For major service update, the Vault Operations team will reach out to product owners for notice.
+Any changes to the Vault Secrets Management tool will be communicated via [#devops-vault](https://chat.developer.gov.bc.ca/channel/devops-vault) and [#devops-alerts](https://chat.developer.gov.bc.ca/channel/devops-alerts) Rocket.Chat channels. For major service updates, the Vault Operations team will reach out to product owners for notice.
 
 ### Service improvements
 
-Vault Secrets Management improvements including system upgrades, feature integration and issue fixing. The Vault Operations team will be conducting the operation on a scheduled time, with advanced notice in the #devops-vault Rocket.Chat channel. If disruption or downtime is expected during service improvement, the team will discuss on maintenance time in the channel to minimize effects.
+Vault Secrets Management improvements include system upgrades, feature integration and issue fixing. The Vault Operations team conduct these operations at scheduled times, with advance notice in the #devops-vault Rocket.Chat channel. If disruption or downtime is expected during service improvement, the team will provide details to help teams prepare and to verify their applications.
 
 ### Service level
-To be determined.
+Enterprise
 
 ### Security reviews
 
-An STRA for Vault has been completed by the Platform Services team.
+A STRA for Vault has been completed by the Platform Services team.
+
+## Alternatives
+If you would prefer to use a different secrets management system, you may use the External Secrets Operator to link your OpenShift namespace to an external service, such as AWS Secrets Manager or Azure Key Vault.  See the [External Secrets Operator documentation](external-secrets.md) for more information.
 
 ---
 Related links:

src/index.md

@@ -102,6 +102,7 @@ demos of their application.
 
 * [Vault getting started guide](docs/secrets-management/vault-getting-started-guide.md)
 * [Vault secrets management](docs/secrets-management/vault-secrets-management-service.md)
+* [External secrets](docs/secrets-management/external-secrets.md)
 
 ### Security and privacy compliance