No custom metrics for Workflows ending in "error" status

[JeanRev]

Hi there,

We use cusom-metrics to detect the termination status of workflows. For the moment, we are limiting ourselves to the workflow statuses “Succeeded” and “Failed”, which works quite well. However, we have noticed that workflows with an "error" status do not seem to produce custom metrics. We are currently using the following setup(which is very similar to the example referenced in the issue #10662):

metrics:
        prometheus:
          - name: failed_gauge
            labels:
              - key: namespace
                value: "{{ workflow.namespace }}"
              - key: workflow
                value: "{{ workflow.labels.app.kubernetes.io/name }}"
            help: Failed gauge for individual workflows
            when: "{{ status  }} == Failed || {{ status }} == Error"

We have tested this by creating an invalid configuration that does not allow the creation of init/main containers.

There must certainly be situations where an erroneous workflow cannot generate metrics (in the event of a major problem with the workflow controller, for example). But in this example, I would still expect the gauge to be updated.

Or I am missing something ?