issue while setting up prometheus metrics in workflowDefaults.

[jollyamit]

Hello, I am trying to setup a workflowDefaults for metrics as described on this page. But its not working as expected. I am getting empty string for label cron_workflow, where-as workflow i am testing with is of type cron-workflow and it does have that label.

workflowDefaults:
spec:
metrics:
prometheus:
- name: workflow_failed_count
counter:
value: "1"
help: Failed workflow counter
labels:
- key: workflow_name
value: "{{ workflow.name }}"
- key: workflow_namespace
value: "{{ workflow.namespace }}"
- key: workflow_duration_seconds
value: "{{ workflow.duration }}"
- key: workflow_created_at
value: "{{ workflow.creationTimestamp }}"
- key: cron_workflow
value: "{{= sprig.default('', workflow.labels['workflows.argoproj.io/cron-workflow']) }}"
when: "{{workflow.status}} == Failed || {{workflow.status}} == Error"

in order to debug further i added few more labels

        - key: cron_workflow
          value: "{{= sprig.default('', workflow.labels['workflows.argoproj.io/cron-workflow']) }}"
        - key: workflow_labels
          value: "{{ workflow.labels }}"
        - key: workflow_labels2
          value: "{{ workflow.labels['workflows.argoproj.io/cron-workflow'] }}"  

and below is what i see in the submitted workflow

      - key: cron_workflow
        value: ''
      - key: workflow_labels
        value: '{"workflows.argoproj.io/cron-workflow":"ml-cron-wf"}'
      - key: workflow_labels
        value: '{{ workflow.labels[''workflows.argoproj.io/cron-workflow''] }}'

value for cron_workflow is substituted with empty string, workflow_labels is showing correct values and for workflow_labels2 its not evaluated and printed as-is. However i do see below error message in the submitted workflow.

- message: >-
    unable to substitute parameters for metric 'workflow_failed_count':
    failed to resolve {{
    workflow.labels['workflows.argoproj.io/cron-workflow'] }}
  status: 'True'
  type: MetricsError

i am using version v3.6.5