Initialise counters on custom metrics to zero

[crileroro]

Hello team,
I have a question regarding custom metrics in workflows, specifically the type count.
Is there a way to initialise a custom count metric to zero in the workflow template? If I use the following snippet code in my template, it initialises the metric to No Data which makes it difficult to detect an initial increase in the metric.

metrics:
    prometheus:
      - name: custom_workflow_result_count
        help: "Count of Workflow results by status"
        labels:
          - key: status
            value: "{{workflow.status}}"
        counter:
          value: "1"

To give you an example of the problem I currently have on AlertManager, I am using the expression increase(argo_workflows_custom_workflow_result_count{status="Failed"}[5m]) > 0
This expression will basically detect any increase in the last 5 minutes. The problem I have is at the first failed workflow. Unfortunately, I will not get notified because the increase() function will output No Data at first and then 0. If the workflow fails again and again (i.e. more than once), it will work as expected because the difference will now be > 0.
How are you handling this type of situations?
As best practice in Prometheus, we should initialise zero values on counters like this. It saves a lot of trouble. (See Guidelines to instrument the code - section Avoid Missing Metrics here)

Thank you for your help!

[ltartarini]

Could be related to the metricsTTL?

#12790