AWS Signer

AWS Signer is a fully managed code-signing service to ensure the trust and integrity of your code. In order to use AWS Signer, you need to create a signing profile on your AWS account:

aws signer put-signing-profile \
     --profile-name container_signing_profile \
     --platform-id Notation-OCI-SHA384-ECDSA \
     --signature-validity-period value=12,type='MONTHS'

You can check the results, you will need the id/arn later

aws signer get-signing-profile --profile-name container_signing_profile

export ID=$(aws signer get-signing-profile --profile-name container_signing_profile | jq -r '.arn')

Notation

Notation is a CLI project to add signatures as standard items in the OCI registry ecosystem, and to build a set of simple tooling for signing and verifying these signatures. To sign the images, you need to have the notation cli tool installed on your system. For MacOS and Linux, you can use brew, or follow the documentation:

brew install notation

To be able to use AWS Signer service with notation, you need to add the plugin as described here.

You can either have the bundled version with notation (like installing rpm, deb, or pkg files), or only download the plugin:

# For MacOS x86
wget https://d2hvyiie56hcat.cloudfront.net/darwin/amd64/plugin/latest/notation-aws-signer-plugin.zip

notation plugin install --file notation-aws-signer-plugin.zip

notation plugin ls
# NAME                                   DESCRIPTION                      VERSION   CAPABILITIES                                                                                             ERROR
# com.amazonaws.signer.notation.plugin   AWS Signer plugin for Notation   1.0.298   [SIGNATURE_GENERATOR.ENVELOPE SIGNATURE_VERIFIER.TRUSTED_IDENTITY SIGNATURE_VERIFIER.REVOCATION_CHECK]   <nil>

Since we will use AWS Signer, we need to add AWS root certificates to notary:

wget https://d2hvyiie56hcat.cloudfront.net/aws-signer-notation-root.cert

notation cert add --type signingAuthority --store aws-signer-ts aws-signer-notation-root.cert
# Successfully added following certificates to named store aws-signer-ts of type signingAuthority:
# aws-signer-notation-root.cert

# AWS GovCloud
wget https://d2hvyiie56hcat.cloudfront.net/aws-us-gov-signer-notation-root.cert

notation cert add --type signingAuthority --store aws-us-gov-signer-ts aws-us-gov-signer-notation-root.cert
# Successfully added following certificates to named store aws-us-gov-signer-ts of type signingAuthority:
# aws-us-gov-signer-notation-root.cert

notation cert ls
# STORE TYPE         STORE NAME             CERTIFICATE
# signingAuthority   aws-signer-ts          aws-signer-notation-root.cert
# signingAuthority   aws-us-gov-signer-ts   aws-us-gov-signer-notation-root.cert

Check all configured files here:

# On MacOS
tree /Users/$(whoami)/Library/Application\ Support/notation/
/Users/korayoksay/Library/Application Support/notation/
├── plugins
│   └── com.amazonaws.signer.notation.plugin
│       ├── LICENSE
│       ├── THIRD_PARTY_LICENSES
│       └── notation-com.amazonaws.signer.notation.plugin
├── trustpolicy.json
└── truststore
    └── x509
        ├── ca
        └── signingAuthority
            ├── aws-signer-ts
            │   └── aws-signer-notation-root.cert
            └── aws-us-gov-signer-ts
                └── aws-us-gov-signer-notation-root.cert

Sign the image and SBOM

Access to the ECR repo

Create the SBOM (Software Bill of Materials) and sign both the image and SBOM:

export AWS_REGION="eu-central-1"
export AWS_ACCOUNT_ID="463470985768"
export ECR_REGISTRY="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
export ECR_NAMESPACE="signer-demo"
export IMAGE_NAME="nginx"
export IMAGE=${ECR_REGISTRY}/${ECR_NAMESPACE}/${IMAGE_NAME}:signed

# If the ECR registry is private, notation needs to login
aws ecr get-login-password --region ${AWS_REGION} | \
    notation login --username AWS --password-stdin ${ECR_REGISTRY}

# Create the SBOM
docker sbom --format spdx-json --output SBOM.json $IMAGE

# Attach the SBOM to the IMAGE
oras attach ${IMAGE} SBOM.json --artifact-type sbom/demo

# Check how it looks
oras discover ${IMAGE} --format=tree

export SBOM_DIGEST=$(oras discover $IMAGE --format=json | jq -r '.manifests[] | select(.artifactType == "sbom/demo") | .digest')
export SBOM=${ECR_REGISTRY}/${ECR_NAMESPACE}/${IMAGE_NAME}@${SBOM_DIGEST}

# Sign the image
notation sign ${IMAGE} --plugin "com.amazonaws.signer.notation.plugin" --id ${ID}
# Successfully signed 463470985768.dkr.ecr.eu-central-1.amazonaws.com/thyris/rag-service@sha256:85c8ca3443c86e3eb9c9f053e6c01870666c5d6c5a8390a47ed88c74077f146d

# Sign the SBOM
notation sign ${SBOM} --plugin "com.amazonaws.signer.notation.plugin" --id ${ID}

# Check again how it looks
oras discover ${IMAGE} --format=tree

Verify the image locally

You need a trust policy:

notation verify ${IMAGE}
# Error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy

Create the trust policy:

cat <<EOF > aws_trust_policy.json
{
   "version":"1.0",
   "trustPolicies":[
      {
         "name":"aws-signer-tp",
         "registryScopes":[
            "*"
         ],
         "signatureVerification":{
            "level":"strict"
         },
         "trustStores":[
            "signingAuthority:aws-signer-ts",
            "signingAuthority:aws-us-gov-signer-ts"
         ],
         "trustedIdentities":[
            "$ID"
         ]
      }
   ]
}
EOF

Import it:

notation policy import aws_trust_policy.json 
# Trust policy configuration imported successfully.

Try to verify it again:

notation verify ${IMAGE}
# Successfully verified signature for 463470985768.dkr.ecr.eu-central-1.amazonaws.com/thyris/rag-service@sha256:85c8ca3443c86e3eb9c9f053e6c01870666c5d6c5a8390a47ed88c74077f146d

Verify the image on Kubernetes using Kyverno

For the demo, start a local minikube cluster (it can also be done on EKS, or any other k8s cluster)

minikube start --cpus=4 --memory 8192

Deploy Kyverno and the kyverno-notation-aws plugin:

# Kyverno
kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.12.7/install.yaml

# kyverno-notation-aws plugin
kubectl apply -f https://raw.githubusercontent.com/nirmata/kyverno-notation-aws/refs/heads/main/configs/install.yaml

# trustpolicy and truststore crds
kubectl apply -f https://raw.githubusercontent.com/nirmata/kyverno-notation-aws/refs/heads/main/configs/crds/notation.nirmata.io_trustpolicies.yaml
kubectl apply -f https://raw.githubusercontent.com/nirmata/kyverno-notation-aws/refs/heads/main/configs/crds/notation.nirmata.io_truststores.yaml

# truststore:
kubectl apply -f https://raw.githubusercontent.com/nirmata/kyverno-notation-aws/refs/heads/main/configs/samples/truststore.yaml

# Wait until Kyverno is up and running:
kubectl wait pods -l app.kubernetes.io/instance=kyverno --for=condition=Ready -n kyverno --timeout=120s

Create the trust policy:

cat <<EOF | kubectl apply -f -
apiVersion: notation.nirmata.io/v1alpha1
kind: TrustPolicy
metadata:
  name: tp-test-notation
spec:
  version: '1.0'
  trustPolicyName: tp-test-notation
  trustPolicies:
  - name: aws-signer-tp
    registryScopes:
    - ${ECR_REGISTRY}/${ECR_NAMESPACE}/${IMAGE_NAME}
    signatureVerification:
      level: strict
      override: {}
    trustStores:
    - signingAuthority:aws-signer-ts
    trustedIdentities:
    - ${ID}
EOF

Create the Kyverno Policy for the signature verification:

cat <<EOF | kubectl apply -f -
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-images     
spec:
  validationFailureAction: Enforce
  failurePolicy: Fail
  webhookTimeoutSeconds: 30
  schemaValidation: false
  rules:
  - name: call-aws-signer-extension
    match:
      any:
      - resources:
          namespaces:
          - test-notation
          kinds:
          - Pod
          operations:
            - CREATE
            - UPDATE
    context:
    - name: tlscerts
      apiCall:
        urlPath: "/api/v1/namespaces/kyverno-notation-aws/secrets/svc.kyverno-notation-aws.svc.tls-pair"
        jmesPath: "base64_decode( data.\"tls.crt\" )"
    - name: response
      apiCall:
        method: POST
        data:
        - key: images
          value: "{{images}}"
        - key: imageReferences
          value: 
          - "${ECR_REGISTRY}*"
        - key: trustPolicy
          value: "tp-{{request.namespace}}"
        - key: attestations
          value: 
          - imageReference: "*"
            type: 
            - name: sbom/demo
              conditions:
                all:
                - key: \{{creationInfo.licenseListVersion}}
                  operator: GreaterThanOrEquals
                  value: "3.16"
                  message: invalid license version
        service:
          url: https://svc.kyverno-notation-aws/checkimages
          caBundle: '{{ tlscerts }}'
    mutate:
      foreach:
      - list: "response.results"
        patchesJson6902: |-
            - path: '{{ element.path }}'
              op: '{{ element.op }}'
              value: '{{ element.value }}'

EOF

Make sure that Kyverno is able to access the private ECR registry. On EKS, this can be resolved by IRSA or pod identidy, but for others registry secret is necessary:

AWS_TOKEN=$(aws ecr get-login-password --region eu-central-1)
kubectl create secret docker-registry regcred \
    --docker-username=AWS \
    --docker-password=${AWS_TOKEN} \
    --docker-server=${ECR_REGISTRY} \
    -n kyverno-notation-aws 

Edit kyverno-notation-aws deployment and add --imagePullSecrets=regcred parameter

kubectl patch deployment kyverno-notation-aws \
    -n kyverno-notation-aws --type='json' \
    -p='[{
          "op": "add", 
          "path": "/spec/template/spec/containers/0/args/-", 
          "value": "--imagePullSecrets=regcred"
        }]'

kubectl wait pods -l app=kyverno-notation-aws \
    --for=condition=Ready -n kyverno-notation-aws --timeout=120s

Same also applies to the AWS Signer access from kyverno-notation-aws, so we need to add AWS credentials:

# $> cat credentials.txt
# AWS_ACCESS_KEY_ID=xxx
# AWS_SECRET_ACCESS_KEY=xxx
kubectl create secret generic aws-credentials \
    --from-env-file=credentials.txt -n kyverno-notation-aws

kubectl patch deployment kyverno-notation-aws \
    -n kyverno-notation-aws --type='json' \
    -p='[{
          "op": "add",
          "path": "/spec/template/spec/containers/0/envFrom", 
          "value": [{"secretRef": {"name": "aws-credentials"}}]
        }]'

kubectl wait pods -l app=kyverno-notation-aws \
    --for=condition=Ready -n kyverno-notation-aws --timeout=120s

Create Pods and Test the Policy

kubectl create ns test-notation

# unsingned image -- expected to fail
kubectl -n test-notation run test-unsigned --image=$ECR_REGISTRY/${ECR_NAMESPACE}/${IMAGE_NAME}:unsigned

# signed image -- expected to be successful
kubectl -n test-notation run test-signed --image=$IMAGE