New commit in git repository not triggering update

[pierskarsenbarg]

What happened?

On occasion (but I seem to be running into it quite a lot at the moment), a new commit to GitHub isn't triggering an update.

Had a chat with Eron yesterday, but some of the statues had changed and it was hard to find some useful information. However, I've run into it again today so I've been collating information.

I've created a gist to contain as much information as possible: https://gist.github.com/pierskarsenbarg/3506bb90321a1a7f1862fab90756584e

Files are as follows:

• flux-source-controller.log: logs from the pod running the Flux source controller
• operator-logs.log: logs from the pulumi container in the pod that's running the PKO controller
• gitrepository.yaml: output of running k get gitrepository -n {namespace} -o yaml
• pulumi-resources.yaml: output of running k get pulumi -n {namespace} -o yaml
• workspace-statefulset.yaml: output of running k get statefulsets/{workspace stateful set} -n {namespace} -o yaml

The repository that I'm updating to trigger the can be found here: https://github.com/pierskarsenbarg/simple-random

The latest commit that was fetched by the Flux source controller was pierskarsenbarg/simple-random@00ca386 (you can see this in the status for the gitrepository resource)

You can see in the operator controller logs that the last update that was run came from the previous commit: https://gist.github.com/pierskarsenbarg/3506bb90321a1a7f1862fab90756584e#file-operator-logs-log-L71-L86

Output of pulumi about

Pulumi CLI: 3.153.1
PKO version: 2.0.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[EronWright]

Here's a watch log showing a possible root cause, that the artifact is updated separately from the Ready status. Since both the "before and after" objects would be considered to be ready, the fluxSourceReadyPredicate would suppress reconciliation of the latter. I effected this by pushing to the repository.

Possible workarounds:

1. compare the Ready message content to the .status.artifact.revision.
2. compare the lastUpdateTime to the lastTransitionTime.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"source.toolkit.fluxcd.io/v1","kind":"GitRepository","metadata":{"annotations":{},"name":"pierskarsenbarg-simple-random","namespace":"default"},"spec":{"interval":"30s","ref":{"branch":"main"},"timeout":"60s","url":"https://github.com/pierskarsenbarg/simple-random"}}
  creationTimestamp: "2025-05-27T20:42:28Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 2
  name: pierskarsenbarg-simple-random
  namespace: default
  resourceVersion: "2292286"
  uid: 6dca94f9-deeb-4713-9b2f-55530b76ee12
spec:
  interval: 30s
  ref:
    branch: main
  timeout: 60s
  url: https://github.com/EronWright/simple-random.git
status:
  artifact:
    digest: sha256:617e0171fd6c8d0b01c04ca9ef788b6447fab174c48411fb9a1cb857a9b4aae5
    lastUpdateTime: "2025-05-27T20:42:30Z"
    path: gitrepository/default/pierskarsenbarg-simple-random/00ca386d43834c41d9626b6d93137d396fd771ed.tar.gz
    revision: main@sha1:00ca386d43834c41d9626b6d93137d396fd771ed
    size: 585
    url: http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pierskarsenbarg-simple-random/00ca386d43834c41d9626b6d93137d396fd771ed.tar.gz
  conditions:
  - lastTransitionTime: "2025-05-27T21:27:13Z"
    message: 'building artifact: new upstream revision ''main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de'''
    observedGeneration: 2
    reason: Progressing
    status: "True"
    type: Reconciling
  - lastTransitionTime: "2025-05-27T21:27:13Z"
    message: 'building artifact: new upstream revision ''main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de'''
    observedGeneration: 2
    reason: Progressing
    status: Unknown
    type: Ready
  - lastTransitionTime: "2025-05-27T20:42:30Z"
    message: stored artifact for revision 'main@sha1:00ca386d43834c41d9626b6d93137d396fd771ed'
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: ArtifactInStorage
  - lastTransitionTime: "2025-05-27T21:27:13Z"
    message: new upstream revision 'main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de'
    observedGeneration: 2
    reason: NewRevision
    status: "True"
    type: ArtifactOutdated
  observedGeneration: 2
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"source.toolkit.fluxcd.io/v1","kind":"GitRepository","metadata":{"annotations":{},"name":"pierskarsenbarg-simple-random","namespace":"default"},"spec":{"interval":"30s","ref":{"branch":"main"},"timeout":"60s","url":"https://github.com/pierskarsenbarg/simple-random"}}
  creationTimestamp: "2025-05-27T20:42:28Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 2
  name: pierskarsenbarg-simple-random
  namespace: default
  resourceVersion: "2292287"
  uid: 6dca94f9-deeb-4713-9b2f-55530b76ee12
spec:
  interval: 30s
  ref:
    branch: main
  timeout: 60s
  url: https://github.com/EronWright/simple-random.git
status:
  artifact:
    digest: sha256:617e0171fd6c8d0b01c04ca9ef788b6447fab174c48411fb9a1cb857a9b4aae5
    lastUpdateTime: "2025-05-27T20:42:30Z"
    path: gitrepository/default/pierskarsenbarg-simple-random/00ca386d43834c41d9626b6d93137d396fd771ed.tar.gz
    revision: main@sha1:00ca386d43834c41d9626b6d93137d396fd771ed
    size: 585
    url: http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pierskarsenbarg-simple-random/00ca386d43834c41d9626b6d93137d396fd771ed.tar.gz
  conditions:
  - lastTransitionTime: "2025-05-27T21:27:13Z"
    message: stored artifact for revision 'main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de'
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-05-27T21:27:13Z"
    message: stored artifact for revision 'main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de'
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: ArtifactInStorage
  observedGeneration: 2
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"source.toolkit.fluxcd.io/v1","kind":"GitRepository","metadata":{"annotations":{},"name":"pierskarsenbarg-simple-random","namespace":"default"},"spec":{"interval":"30s","ref":{"branch":"main"},"timeout":"60s","url":"https://github.com/pierskarsenbarg/simple-random"}}
  creationTimestamp: "2025-05-27T20:42:28Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 2
  name: pierskarsenbarg-simple-random
  namespace: default
  resourceVersion: "2292288"
  uid: 6dca94f9-deeb-4713-9b2f-55530b76ee12
spec:
  interval: 30s
  ref:
    branch: main
  timeout: 60s
  url: https://github.com/EronWright/simple-random.git
status:
  artifact:
    digest: sha256:c84370fd81474ea743d030a86b74b847c714af74d87b038de5fa41d21336e0e6
    lastUpdateTime: "2025-05-27T21:27:13Z"
    path: gitrepository/default/pierskarsenbarg-simple-random/9eb98f3e74333ae8abadf0d73356a1bca9d3b9de.tar.gz
    revision: main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de
    size: 585
    url: http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pierskarsenbarg-simple-random/9eb98f3e74333ae8abadf0d73356a1bca9d3b9de.tar.gz
  conditions:
  - lastTransitionTime: "2025-05-27T21:27:13Z"
    message: stored artifact for revision 'main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de'
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-05-27T21:27:13Z"
    message: stored artifact for revision 'main@sha1:9eb98f3e74333ae8abadf0d73356a1bca9d3b9de'
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: ArtifactInStorage
  observedGeneration: 2

[EronWright]

I've identified the root cause and filed an issue in the upstream Flux repository:
fluxcd/pkg#934

I'll seek to use a workaround, e.g.

1. Check for equality between .status.artifact.lastUpdateTime and .status.conditions[type=Ready].lastTransitionTime.
2. Check for .status.artifact.revision within .status.conditions[type=Ready].message.