argo wait hangs forever even though the workflow has Succeeded and its Pod `Completed

[Dentrax]

Not sure whether this is a bug or if I'm missing something, wanted to start a discussion instead of filing Bug Report issue.

$ k get workflow argo-wf-test-workflow-zp7g5
NAME                                                STATUS      AGE   MESSAGE
argo-wf-test-workflow-zp7g5   Succeeded   26m

This hangs forever:

argo -v wait argo-wf-test-zp7g5
DEBU[2025-02-21T11:29:14.738Z] CLI version                                   version="{v3.6.4+77552d8.dirty 2025-02-10T13:54:30Z 77552d852da5c53a1d8c9918b3474d8bff06260a v3.6.4 dirty go1.23.6 gc darwin/arm64}"
DEBU[2025-02-21T11:29:14.739Z] Client options                                opts="(argoServerOpts=(url=,path=,secure=true,insecureSkipVerify=false,http=false),instanceID=)"
I0221 11:29:14.740490   99631 loader.go:395] Config loaded from file:  /Users/furkan/.kube/cluster1
I0221 11:29:14.741123   99631 loader.go:395] Config loaded from file:  /Users/furkan/.kube/cluster1
I0221 11:29:15.207077   99631 round_trippers.go:553] GET https://<REDACTED>/apis/argoproj.io/v1alpha1/namespaces/argo-events/workflows/argo-wf-test-zp7g5 200 OK in 464 milliseconds
I0221 11:29:15.274673   99631 round_trippers.go:553] GET <REDACTED>/apis/argoproj.io/v1alpha1/namespaces/argo-events/workflows?fieldSelector=metadata.name%3Dargo-wf-test-zp7g5&labelSelector=%21workflows.argoproj.io%2Fcontroller-instanceid&resourceVersion=0&watch=true 200 OK in 48 milliseconds
DEBU[2025-02-21T11:29:15.274Z] Piping events to channel

Those also hangs:

kubectl wait --for=condition=Succeeded --timeout=300s -n argo-events workflow/argo-wf-test-zp7g5

kubectl wait --for=condition=Completed --timeout=300s -n argo-events workflow/argo-wf-test-zp7g5

kubectl wait --for=condition=Ready --timeout=300s -n argo-events workflow/aargo-wf-test-zp7g5

$ argo version
argo: v3.6.4+77552d8.dirty
  BuildDate: 2025-02-10T13:54:30Z
  GitCommit: 77552d852da5c53a1d8c9918b3474d8bff06260a
  GitTreeState: dirty
  GitTag: v3.6.4
  GoVersion: go1.23.6
  Compiler: gc
  Platform: darwin/arm64

Events image:

quay.io/argoproj/argo-events:v1.9.5

argo get argo-wf-test-zp7g5 -o yaml

Name: argo-wf-tbp-test-zp7g5
Namespace: argo-events
ServiceAccount: unset (will run with the default ServiceAccount)
Status: Succeeded
Conditions:
Completed True
Created: Fri Feb 21 10:53:26 +0300 (33 minutes ago)
Started: Fri Feb 21 10:53:26 +0300 (33 minutes ago)
Finished: Fri Feb 21 10:53:26 +0300 (33 minutes ago)
Duration: 0 seconds
Progress: 0/0
Parameters:
alerts: []
alertname: MyTestAlert

STEP TEMPLATE PODNAME DURATION MESSAGE
✔ argo-wf-tbp-test-zp7g5 argo-wf-test
└───○ argo-wf-test argo-wf-test/argo-wf-test when 'MyTestAlert == "MyTestAlert2"' evaluated false

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
annotations:
  workflows.argoproj.io/pod-name-format: v2
creationTimestamp: "2025-02-21T07:53:26Z"
generateName: argo-wf-test-workflow-
generation: 2
labels:
  events.argoproj.io/action-timestamp: "1740124406616"
  events.argoproj.io/sensor: argo-wf-test
  events.argoproj.io/trigger: argo-wf-test-workflow-trigger
  workflows.argoproj.io/completed: "true"
  workflows.argoproj.io/phase: Succeeded
name: argo-wf-test-workflow-zp7g5
namespace: argo-events
resourceVersion: "685047947"
uid: dfbeafb0-a8e3-4074-b71b-1ad6b99307ce
spec:
activeDeadlineSeconds: 28800
arguments:
  parameters:
  - name: alerts
  - name: alertname
    value: MyTestAlert
entrypoint: argo-wf-test
imagePullSecrets:
- name: ty-docker-registry
podGC: {}
templates:
- inputs:
    parameters:
    - name: alerts
    - name: alertname
  metadata: {}
  name: argo-wf-test
  outputs: {}
  steps:
  - - arguments:
        parameters:
        - name: alerts
          value: '{{inputs.parameters.alerts}}'
        - name: alertname
          value: '{{inputs.parameters.alertname}}'
      name: argo-wf-test
      templateRef:
        name: argo-wf-test
        template: argo-wf-test
      when: '{{inputs.parameters.alertname}} == "MyTestAlert"'
ttlStrategy:
  secondsAfterCompletion: 86400
status:
artifactGCStatus:
  notSpecified: true
artifactRepositoryRef:
  artifactRepository:
    oss:
      accessKeySecret:
        key: accessKey
        name: test-credentials
      bucket: test
      endpoint: <REDACTED>
      secretKeySecret:
        key: secretKey
        name: test-credentials
  configMap: artifact-repositories
  key: default-oss-artifact-repository
  namespace: argo-events
conditions:
- status: "True"
  type: Completed
finishedAt: "2025-02-21T07:53:26Z"
nodes:
  argo-wf-test-workflow-zp7g5:
    children:
    - argo-wf-test-workflow-zp7g5-1757136186
    displayName: argo-wf-test-workflow-zp7g5
    finishedAt: "2025-02-21T07:53:26Z"
    id: argo-wf-test-workflow-zp7g5
    inputs:
      parameters:
      - name: alerts
      - name: alertname
        value: ScheduleDisabledAndReadyNode
    name: argo-wf-test-workflow-zp7g5
    outboundNodes:
    - argo-wf-test-workflow-zp7g5-2985485489
    phase: Succeeded
    startedAt: "2025-02-21T07:53:26Z"
    templateName: argo-wf-test
    templateScope: local/argo-wf-test-workflow-zp7g5
    type: Steps
  argo-wf-test-workflow-zp7g5-1757136186:
    boundaryID: argo-wf-test-workflow-zp7g5
    children:
    - argo-wf-test-workflow-zp7g5-2985485489
    displayName: '[0]'
    finishedAt: "2025-02-21T07:53:26Z"
    id: argo-wf-test-workflow-zp7g5-1757136186
    name: argo-wf-test-workflow-zp7g5[0]
    phase: Succeeded
    startedAt: "2025-02-21T07:53:26Z"
    templateScope: local/argo-wf-test-workflow-zp7g5
    type: StepGroup
  argo-wf-test-workflow-zp7g5-2985485489:
    boundaryID: argo-wf-test-workflow-zp7g5
    displayName: argo-wf-test
    finishedAt: "2025-02-21T07:53:26Z"
    id: argo-wf-test-workflow-zp7g5-2985485489
    message: when 'MyTestAlert2 == "MyTestAlert"'
      evaluated false
    name: argo-wf-test-workflow-zp7g5[0].argo-wf-test
    phase: Skipped
    startedAt: "2025-02-21T07:53:26Z"
    templateRef:
      name: argo-wf-test
      template: argo-wf-test
    templateScope: local/argo-wf-test-workflow-zp7g5
    type: Skipped
phase: Succeeded
progress: 0/0
startedAt: "2025-02-21T07:53:26Z"
storedTemplates:
  namespaced/argo-wf-test/argo-wf-test:
    container:
      args:
      - 'echo ''Alerts: {{inputs.parameters.alerts}}''; echo ''Alertname: {{inputs.parameters.alertname}}'';
        sleep 60'
      command:
      - sh
      - -c
      image: busybox
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: alerts
      - name: alertname
    metadata:
      annotations:
        argo-hub-template/description: Test
    name: argo-wf-test
    outputs: {}
    serviceAccountName: tbp-notification-forwarder-template

[Dentrax]

It seems it was related to API server address, I was using FQDN but switching back to IP make this work. Closing this one. Sorry for inconveniences.