Motoko value tracker for prometheus

Overview

The purpose of PromTracker is to record different kinds of values during the operation of a canister, aggregate and in some cases process them, and export them in the Prometheus exposition format.

The exposition format can then be provided by an http endpoint from where it can be accessed by a scraper. Serving the http endpoint is not directly part of this package, but is provided in the example directory.

The two main value types that can be tracked are CounterValue and GaugeValue. These values are explicitly updated by events in canister code.

A counter is normally an ever-increasing counter such as the number of total requests received. The scraper only sees its last value. The values between scraping events are considered not important.

A gauge is a more frequently changing and normally fluctuating value such as the size of the last request, time between last two events, etc. The values between scraping events are considered important. That's why a gauge value allows to automatically track the high and low watermark between scraping events as well as a histogram. It also allowst to export histograms taken over time can be used to create heatmaps.

The third value type is the PullValue which is stateless version of a counter. It is not explicitly updated by events in canister code. Instead, the value is calculated on the fly when the scraping happens. This type is convenient for exposing a canister's system state such a cycle balance and memory size because those are already tracked by the runtime or management canister and canister code does not need to update them explicitly. This type can also be used to expose an expression in one or more other tracked values regardless of those values' types such as for example the sum of two other values.

The tracker class PromTracker is instantiated once per canister. Then various code modules can each register a value with the PromTracker class that they like to have tracked and that they the maintain (i.e. update). The http endpoint accesses only the PromTracker instance, not the various code modules, to get the exposition of all tracked values.

Links

The package is published on MOPS and GitHub.

The API documentation can be found here on Mops.

For updates, help, questions, feedback and other requests related to this package join us on:

• OpenChat group
• Twitter
• Dfinity forum

Examples

Canister including http endpoint

Our example canister tracks the "heartrate" by tracking the time in milliseconds between subsequent heartbeats. This value is a GaugeValue and it allows us to see the high and low watermarks as well as the distribution in the form of a histogram.

The PromTracker class

Create tracker instance like this:

let tracker = PT.PromTracker(65);

65 seconds is the recommended interval if prometheus pulls stats with interval 60 seconds. This value used to clear high and low watermarks in gauge values, so each highest and lowest value during your canister lifecycle will be reflected in the prometheus data.

Add some values:

let successfulHeartbeats = tracker.addCounter("successful_heartbeats", "", true);
let failedHeartbeats = tracker.addCounter("failed_heartbeats", "", true);
let heartbeats = tracker.addPullValue("heartbeats", "", func() = successfulHeartbeats.value() + failedHeartbeats.value());
let heartbeatDuration = tracker.addGauge("heartbeat_duration", "", #both, null, false);

Update values:

successfulHeartbeats.add(2);
failedHeartbeats.add(1);
heartbeatDuration.update(10);
heartbeatDuration.update(18);
heartbeatDuration.update(14);

Get prometheus exposition:

let text : Text = tracker.renderStats();

Make stats surviving canister upgrades:

stable var statsData : PT.StableData = null;

system func preupgrade() {
  statsData := tracker.share();
};

system func postupgrade() {
  tracker.unshare(statsData);
};
  

PullValue

A stateless value interface, which runs the provided getter function on demand.

let storageSize = tracker.addPullValue("storage_size", "", func() = storage.size());

CounterValue

An accumulating counter value interface. Second argument is a flag whether you want to save the state of this value to stable data using share/unshare api

    let requestsAmount = tracker.addCounter("requests_amount", "", false);
    // now it will output 0
    requestsAmount.add(3);
    // it will output 3
    requestsAmount.add(1);
    // now it will output 4
    requestsAmount.set(0);
    // now it will output 0 again

GaugeValue

A gauge value interface for ever-changing value, with ability to catch the highest and lowest value during interval, set on tracker instance and ability to bucket the values for histogram output. Outputs few stats at once: sum of all pushed values, amount of pushes, lowest value during interval, highest value during interval, histogram buckets. 4th argument accepts edge values for buckets

    let requestDuration = tracker.addGauge("request_duration", "", #both, ?[50, 110], false);
    requestDuration.update(123);
    requestDuration.update(101);
    // now it will output stats: 
    // request_duration_sum: 224
    // request_duration_count: 2
    // request_duration_high_watermark: 123
    // request_duration_low_watermark: 101
    // request_duration_low_watermark: 101
    // request_duration_bucket{le="50"}: 0
    // request_duration_bucket{le="110"}: 1
    // request_duration_bucket{le="+Inf"} 2

System metrics

PromTracker has the ability to extend your prometheus exposition output with these pull values:

1. cycles_balance // Cycles.balance()
2. rts_memory_size // Prim.rts_memory_size()
3. rts_heap_size // Prim.rts_heap_size()
4. rts_total_allocation // Prim.rts_total_allocation()
5. rts_reclaimed // Prim.rts_reclaimed()
6. rts_max_live_size // Prim.rts_max_live_size()
7. rts_max_stack_size // Prim.rts_max_stack_size()
8. rts_callback_table_count // Prim.rts_callback_table_count()
9. rts_callback_table_size // Prim.rts_callback_table_size()
10. rts_mutator_instructions // Prim.rts_mutator_instructions()
11. rts_collector_instructions // Prim.rts_collector_instructions()
12. rts_upgrade_instructions // Prim.rts_upgrade_instructions()
13. rts_stable_memory_size // Prim.rts_stable_memory_size()
14. rts_logical_stable_memory_size // Prim.rts_logical_stable_memory_size()
15. canister_version // Prim.canisterVersion()

To register them, call function:

metrics.addSystemValues();

Copyright

MR Research AG, 2023-25

Authors

Andy Gura (AndyGura) with contributions from Timo Hanke (timohanke)

License

Apache-2.0