Histogram bucket range.

[Roiocam]

I use histograms to record the latency distribution of my application.

When the traffic was peaking, the backpressure kicks in and blocks messages in the message queue (MQ). As a result, the time that messages are blocked in the MQ increases significantly. This makes it difficult for me to gain insights into the specific latency of these spikes in Grafana and Prometheus.

I have noticed that OTEL's exponential histograms are already stable versions, but I am unsure if exponential histograms can automatically adapt to changes in user latency and find the appropriate resolution. Or I move towards to the wrong way?

[jack-berg]

Exponential histograms do automatically adapt to the range of measurements they represent. Each one has a scale parameter, a maximum number of buckets, and an index offset parameter. The scale is used to compute the bucket boundaries for any index. The basic idea is that the implementation can compute the maximum scale (scale is inversely related to bucket size, so a larger scale equals smaller buckets) given the range of measurements and the maximum number of buckets. The scale starts high (i.e. small buckets), and adjusts to become smaller (i.e. rescales) when a measurement is recorded that it can't accommodate given its scale and the number of buckets it has. When a rescale occurs, the measurements recorded in the previously smaller buckets are redistributed into the now larger buckets. The bucket boundaries are arranged in way that rescaling can't result in any bucket counts being inaccurate by a property called "perfect subsetting".

I know that prometheus has some sort of support for exponential bucket histograms, but the prometheus exporter for the java SDK currently does not support them. Maybe the collector's prometheus exporter supports them thought? If so, you could export them with something like: sdk -> (otlp) -> collector -> (prometheus) -> prometheus?