[BUG] Reloader fails to reload pods on first ConfigMap update by trust-manager unless --reload-on-create=true is set

[hznor]

Describe the bug
Reloader doesn't reload pod associated with a configMap when the configMap is being modified for the first time by trust-manager, unless --reload-on-create=true is defined.
Though, it reloads pods without issues for subsequent values overwrites by trust manager, only the first modification on the configMap by trust-manager fails to trigger a reload by Reloader.

To Reproduce
$ kubectl apply an empty configMap, then make trust-manager override this configMap with new values

Expected behavior
Reloader should detect a change in the configMap as soon as trust manager modifies it

Environment

• Kubernetes/OpenShift Version: v1.31.7

Additional context

At first we kubectl apply an empty configMap:

$ kubectl get cm ca-bundle-exploration-externals -o yaml
apiVersion: v1
data:
  ca-externals.crt: ' '
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"ca-externals.crt":" "},"kind":"ConfigMap","metadata":{"annotations":{},"name":"ca-bundle-exploration-externals","namespace":"default"}}
  creationTimestamp: "2025-07-29T13:39:47Z"
  name: ca-bundle-exploration-externals
  namespace: default
  resourceVersion: "17921903"
  uid: fae9a394-7ec5-4be4-9745-80920443c28c

Then we trigger trust-manager, which overrides the empty configMap with new values (reload doesn't happen here, unless --reload-on-create=true is passed):

$ kubectl get cm ca-bundle-exploration-externals -o yaml
apiVersion: v1
data:
  ca-externals.crt: |-
    -----BEGIN CERTIFICATE-----
[EDITED]
[CERT A]
[EDITED]
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"ca-externals.crt":" "},"kind":"ConfigMap","metadata":{"annotations":{},"name":"ca-bundle-exploration-externals","namespace":"default"}}
    trust.cert-manager.io/hash: cd98349a90a704071cb8ab855c5e6509a365ecd32f70637a8c3ba8299833ce35
  creationTimestamp: "2025-07-29T13:39:47Z"
  labels:
    app.kubernetes.io/component: exploration
    trust.cert-manager.io/bundle: ca-bundle-exploration-externals
  name: ca-bundle-exploration-externals
  namespace: argonos-explo
  ownerReferences:
  - apiVersion: trust.cert-manager.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Bundle
    name: ca-bundle-exploration-externals
    uid: 632f81ac-0595-496f-a80c-e180d38d61ec
  resourceVersion: "17925074"
  uid: fae9a394-7ec5-4be4-9745-80920443c28c

We trigger trust-manager with different input value to get new values in the configMap : reload happens here (with ot without --reload-on-create=true)

$ kubectl get cm ca-bundle-exploration-externals -o yaml
apiVersion: v1
data:
  ca-externals.crt: |-
    -----BEGIN CERTIFICATE-----
[EDITED]
[CERT B]
[EDITED]
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"ca-externals.crt":" "},"kind":"ConfigMap","metadata":{"annotations":{},"name":"ca-bundle-exploration-externals","namespace":"default"}}
    trust.cert-manager.io/hash: 32e4392fe99a33eea980b2eba3cb18c8b1f6eee727bc8c3ec190725a5997bba2
  creationTimestamp: "2025-07-29T13:39:47Z"
  labels:
    app.kubernetes.io/component: exploration
    trust.cert-manager.io/bundle: ca-bundle-exploration-externals
  name: ca-bundle-exploration-externals
  namespace: argonos-explo
  ownerReferences:
  - apiVersion: trust.cert-manager.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Bundle
    name: ca-bundle-exploration-externals
    uid: 632f81ac-0595-496f-a80c-e180d38d61ec
  resourceVersion: "17926470"
  uid: fae9a394-7ec5-4be4-9745-80920443c28c

• Even though the resource UID doesn't change, which indicates that the resource was not deleted by trust-manager, --reload-on-create=true is necessary for a reload to happen.

[msafwankarim]

Please share the workload yaml

[hznor]

Thank you for you response @msafwankarim, I'm not sure to understand which yaml file you are asking for?
I mean the 3 yaml you see in my post above are 3 different versions of the configMap that should trigger a reload.
I cannot share the input certificates I have used with trust-manager, but any x509 certificates will produce the same result.

• In order to install trust-manager:
  https://cert-manager.io/docs/trust/trust-manager/installation/#2-install-cert-manager-optional

• Here is the yaml file of the original "empty" configMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ca-bundle-exploration-externals
data:
  ca-externals.crt: ' '

• Here is the yaml file of the trust-manager Bundle resource:

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: ca-bundle-exploration-externals
  namespace: cert-manager
spec:
  sources:
    - useDefaultCAs: false
    - configMap:
        name: "ca-exploration-webhooks"
        key: ca.crt
  target:
    configMap:
      key: "ca-externals.crt"
      metadata:
        labels:
          [REDACTED]: [REDACTED]
    namespaceSelector:
      matchLabels:
        [REDACTED]: [REDACTED]

[msafwankarim]

I meant the deployment yaml that is using this ConfigMap

[hznor]

Ok @msafwankarim here is the workload yaml definition

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: workload
  annotations:
    configmap.reloader.stakater.com/reload: "ca-bundle-exploration-externals"
spec:
  selector:
    matchLabels:
      app: busybox
  replicas: 1
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - name: busybox 
          image: busybox
          args: [
            "sleep",
            "1d",
          ]
          volumeMounts:
           - mountPath: /usr/local/share/ca-certificates/exploration-externals-ca.crt
             name: ca-bundle-exploration-externals
             subPath: ca-externals.crt
          securityContext:
            seccompProfile:
              type: "RuntimeDefault"
            runAsNonRoot: true
            runAsUser: 999
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
      volumes:
      - configMap:
          defaultMode: 420
          name: ca-bundle-exploration-externals
        name: ca-bundle-exploration-externals

[msafwankarim]

Are you using reloader with namespace/label selector or with namespace scope? Please share reloader's deployment too if possible. trust-manager creates the configmap in every namespace. So this case can only be possible if initial confimap was created in a namespace which is not being watched and when trust manager created a bundle it created a configmap in all namespaces so it triggered create event instead of update event

[hznor]

@msafwankarim as far as I know trust-manager creates a configMap in a given namespace
The given namespace had the reloader label on it

[msafwankarim]

Ok makes sense I just saw the bundle yaml and it had namespaceSelector defined in it.

But the configmap that you initially created shows namespace as default. Is your bundle's namespace selector targetting the default one? Please share the reloader's deployment if possible, I just want to see the container arguments and reloaer related environment variables.

I was able to produce this issue when I was running reloader with namespace scope.