SOLR-17628: Add query quantiles metrics to prometheus endpoint

[jkmuriithi]

https://issues.apache.org/jira/browse/SOLR-17628

Description

Modify the implementation of SolrPrometheusFormatter.exportTimer to export a Prometheus summary containing quantile information instead of a single Prometheus gauge. Rename the Timer-based metrics solr_metrics_core_average_request_time and solr_metrics_core_average_searcher_warmup_time to reflect this change. Remove the solr_metrics_core_requests_time Counter metric.

Solution

Prior to this change, Dropwizard Timer metrics (used for core request handlers and searchers) were exported in Prometheus format as single gauges representing the mean of all observations. This PR replaces the existing mean gauge metrics with a summary that includes quantile metrics, the count (number) of observations, and the sum of all observations.

Sample old output:

# TYPE solr_metrics_core_average_request_time gauge
solr_metrics_core_average_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/file",replica="replica_n1",shard="shard1"} 0.0
solr_metrics_core_average_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/luke",replica="replica_n1",shard="shard1"} 0.0

Sample new output:

# TYPE solr_metrics_core_request_time summary
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/file",replica="replica_n1",shard="shard1",quantile="0.5"} 0.0
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/file",replica="replica_n1",shard="shard1",quantile="0.75"} 0.0
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/file",replica="replica_n1",shard="shard1",quantile="0.99"} 0.0
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/file",replica="replica_n1",shard="shard1",quantile="0.999"} 0.0
solr_metrics_core_request_time_count{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/file",replica="replica_n1",shard="shard1"} 0
solr_metrics_core_request_time_sum{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/file",replica="replica_n1",shard="shard1"} 0.0
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/luke",replica="replica_n1",shard="shard1",quantile="0.5"} 0.0
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/luke",replica="replica_n1",shard="shard1",quantile="0.75"} 0.0
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/luke",replica="replica_n1",shard="shard1",quantile="0.99"} 0.0
solr_metrics_core_request_time{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/luke",replica="replica_n1",shard="shard1",quantile="0.999"} 0.0
solr_metrics_core_request_time_count{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/luke",replica="replica_n1",shard="shard1"} 0
solr_metrics_core_request_time_sum{category="ADMIN",collection="example-collection",core="core_example-collection_shard1_replica_n1",handler="/admin/luke",replica="replica_n1",shard="shard1"} 0.0

Tests

I updated MetricsHandlerTest and SolrPrometheusFormatterTest to align with the changes to exportTimer. ./gradlew test passes on my local machine.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[mlbiscoc]

Thanks @jkmuriithi for doing this!

@dsmiley maybe you can help review this and hopefully agree this is worth adding? I made this jira because I think this is important missing piece of metrics for prometheus and this PR addresses my poor implementation exporting timer (Should be a summary not a gauge average) but also adds support for creating a summary metric type.

The prometheus exporter doesn't seem to support histograms or summaries so this gets ahead of that curve.

[github-actions]

This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the dev@solr.apache.org mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution!

[github-actions]

This PR is now closed due to 60 days of inactivity after being marked as stale. Re-opening this PR is still possible, in which case it will be marked as active again.

[mlbiscoc]

I was looking at this PR for and just remembered I completely forgot about this PR! I think there is still value in having this for at least the 9x branch and then completely removed in 10 with OTEL. Not only did this add missing quantiles, it fixes up the timer export implementation with a summary giving better timer data.

@dsmiley If you have no other comments I'd like to merge this in. Is it too late for 9.9? Not sure where that process went.

[dsmiley]

Until there is a release branch for release X, no PR is "too late" for release X. Once a release branch exists (e.g. branch_9_9), you can take up the question with whoever volunteered to be the release manager (Houston volunteered for 9.9).

[mlbiscoc]

Thanks @jkmuriithi for fixing the precommit failure and merging in main. Could you add a entry into CHANGES.txt under 9.9? I think this belongs under improvement section. Maybe something like "Export metrics timers to wt=prometheus as Prometheus Summaries (introduces count, sum, and quantiles)"

[dsmiley]

+1 thanks for the before & after in JIRA. Really important for showing what changed.