Regression : Missing collectd_mcac_micros_bucket_ Metric in Cassandra 4.1

[Nassz]

Description:

With the release of Cassandra 4.1, we have noticed a regression in the metric collection process for the collectd_mcac_micros_bucket_ metric, which is essential for monitoring and performance analysis. The MCAC module is no longer generating this metric as expected.

Steps to Reproduce:

Use Cassandra version 4.1+.
curl http://localhost:9103/metrics | grep micros_

Expected Behavior:

The collectd_mcac_micros_bucket_ metric should be generated and updated as before in Cassandra 4.1, allowing proper monitoring.

Environment:

• Cassandra Version: 4.1.0
• MCAC version : 0.3.4

[RomainAnselin]

Just for adding information, this issue is still present with more recent Cassandra 4.1 (tested 4.1.5) and the MCAC 0.3.4-beta1

I captured the following error in the Cassandra 4.1 logs which looks like it's falling in the ballpark of the issue:

INFO [insights-6-1] 2024-09-17 15:15:32,037 NoSpamLogger.java:105 - Not able to get buckets for org.apache.cassandra.metrics.dropped_message.internal_dropped_latency.finalize_propose_msg 128 type org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot

There's an existing partial fix in k8ssandra for metrics for this in k8ssandra/management-api-for-apache-cassandra#428 as it seems there's a change in the bucket offsets in Cassandra 4.1, but it doesn't appear to be sufficient as per k8ssandra/management-api-for-apache-cassandra#444 and k8ssandra/management-api-for-apache-cassandra#500

The port for existing fix based on a quick research of the EstimatedHistogramReservoirSnapshot would likely go here in MCAC

metric-collector-for-apache-cassandra/src/main/java/com/datastax/mcac/UnixSocketClient.java

Line 992 in 1392267

if (snapshotClass.contains("EstimatedHistogramReservoirSnapshot"))

[MintakaB]

Same issue