nginx_ingress_controller_ssl_expire_time_seconds Prometheus metrics giving duplicate results for same domain

[Laveshsp]

What happened:

I have an ingress domain that's using tls cert pointing to k8s secret named tls-ingress-secret. It started giving me the cert expiry alert set via prometheus query as below and this was expected.
0 <= floor((avg(nginx_ingress_controller_ssl_expire_time_seconds{cluster='{{ $k }}'}) by (host secret_name, namespace) - time()) / 86400)

Action: I went ahead and deleted the old cert having name tls-ingress-secret. With this my ingress domain started pointing to nginx's default-cert (in ingress namespace) which is expected again. Later I created the k8s secret with same name tls-ingress-secret (in prometheus namespace) having renewed creds. I later also confirmed on browser too that it has picked new certs with new export date that is around 1year from now.

Issue: After completing the above action too, the prometheus query is giving two results for same domain one with default-cert and one more with tls-ingress-secret as shown in below ScreenShot

What you expected to happen:

As soon as the new secret was created, prometheus query should not give results containing the default-cert details and be able to only pick the new cert details and omit the default-cert detail.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

Kubernetes version (use kubectl version): 1.30

Environment: AWS EKS Cluster

• Cloud provider or hardware configuration: AWS EKS Cluster

How to reproduce this issue:

• Install ingress nginx controller from helm chart V1.12.1 on ingress namespace having a custom default-cert configured on same ns
• Create one more k8s tls secret with name tls-ingress-secret on app ns to reference it in below test ingress
• Create any test k8s ingress conf say for eg: test.prom.com by pointing the tls section to tls-ingress-secret
• Run the prometheus query to get one entry for test.prom.com domain pointing to secret tls-ingress-secret (Expected behavior)
• Delete the k8s secret tls-ingress-secret
• Run the prometheus query to get one entry for test.prom.com domain pointing to secret default-cert (Expected behavior)
• Recreate the tls-ingress-secret again on app ns.
• Run the prometheus query again. Here you will get two entries, one with default-cert secret and one more with tls-ingress-secret o same domain(Not expected behavior behavior)

Anything else we need to know:
Request to please assist as this is creating a lot of false alerts wrt tls cert expiry.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.