(cont'd #2174...) Service collector should be able to ignore Disabled services

[bsamarc]

Problem Statement

(This is a continuation of issue #2174 which was closed prior to having all of the details needed)

With the previous WMI implementation it was easy to ignore disabled services with a clause that included the StartMode <> 'Disabled' filter in the query. With the refactor to not use WMI we've lost the ability to limit collection to only collect services that are not disabled.

How do you use relabel to drop this metric:

windows_service_state{name="dcsvc",state="stopped"} 1

but only where

windows_service_start_mode{name="dcsvc", start_mode="disabled"} 1

(start_mode and state are stored under different metrics)

With the previous integration, this would have been handled with
where_clause = "(name = 'dcsvc') AND StartMode <> 'Disabled'"

Proposed Solution

Implement a new CLI flag/ configuration that would allow for a subset of StartModes to be collected (--collector.service.start_modes = 'boot|manual|auto|system' or just --collector.service.collect_disabled = false)

Additional information

Another solution would be to add the start_mode as a label on the other metrics, however that seems like a poor pattern to follow when there's another metric for start_mode.

Acceptance Criteria

No response

[jkroepke]

Whats the use-case or what is the issue with having metrics from disabled services?

[bsamarc]

Windows exporter is deployed to multiple cloned nodes of an application and web farm.

• Node 1 has application services 1 - 10 installed, service 1 is enabled and running while others are disabled and stopped
• Node 2 through 5 have services 1 - 10 installed, services 2 - 10 are enabled and running while service 1 is disabled and stopped
• If Node 1 goes offline, Node 2 enables app service 1
• I alert when windows_service_state{name=~"app_service.*", state="stopped"} = 1, ignoring nodes where the service is disabled

Currently (or, previously), I'm able to deploy my config with
where_clause = "(name LIKE 'app_service%') AND StartMode <> 'Disabled'"
to the entire farm without knowing which node is currently hosting app service 1

---

While I could solve my alerting issue by joining metrics in my query, I also provide Windows dashboards listing the state of the monitored services. In the example above, the state of app service 1 will only display when Node 1 is selected, rather than showing it as Stopped on Nodes 2 through 5

[jkroepke]

Join the metrics like

windows_service_state{name=~"app_service.*", state="stopped"} + on(name) group_left(start_mode) (windows_service_start_mode{start_mode="disabled"} == 0) == 1

should work for you, even on dashboards?

[bsamarc]

Sure, there are workarounds. However, this means rethinking our original design of "Any monitored service should be in a RUNNING state" (e.g. if a service is providing data to Prometheus, it is considered MONITORED, all others are not important)

Also, as we use Grafana Cloud to store our metrics, this impact our metrics usage costs. We do not want to collect and store data for a service which is DISABLED as it's irrelevant to our production status.