feat(metrics): add metrics collection mechanism

[leo-marble]

This PR introduces the concept of a metric collector, a new use case that runs periodically to launch all collectors. It introduces the notion of a Collector which has a list of collectors (both global and organization-specific), a collector is a module which implement how to compute/extract a metric.

A versioning system is implemented within the Collector to determine the list of collectors we are likely to receive and to know how to parse the results accordingly.

Now that the mechanism is implemented, the next steps will be to define the list of metrics and implement the collectors in usecases/metrics_collection, following the example of the stub.

This PR also introduces the concept of periodicity to determine if the metric is monthly (calendar-based), daily, or instantaneous. If there is an overlap between two periods, the collector will handle the separation and emit two items. A utility is provided to manage these date notions.

There are still some TODOs remaining, and there are PRs branching from this one to implement these TODOs. Some configuration still needs to be set up.

[linear]

MAR-1129 Define the CollectData logic

[leo-marble]

cf: #1074 -> Introduce the watermark to bound the time range

[apognu]

Notes from our discussion :

• Maybe, for collection reports with a license key, check that the license key is valid.
• Outside the scope of this PR, but think about adding some safeguards, since those are unauthenticated requests (rate limiting on the Load Balancer?).

[Pascal-Delange]

I read some of the PR, though I'm not finished. I'll try to finish tomorrow.

[Pascal-Delange]

Side question btw, you did the dataset and table creation on BQ manually ?

[apognu]

Side question btw, you did the dataset and table creation on BQ manually ?

For now, yes, manually in the sandbox account. Once we end up on a finalized schema, I will write the Terraform for it.

This is what the table might look like after the few changes we mentioned:

create table `marble-dev-sandbox.metrics.events` (
    start_time timestamp not null,
    end_time timestamp not null,
    deployment_id string(36) not null,
    license_key string(36),
    org_id string(36),
    event_type string not null,
    value float64,
    text string,
    collected_at timestamp default current_timestamp()
)
partition by timestamp_trunc(start_time, month)
cluster by deployment_id, license_key, org_id, event_type;

Partition and clustering keys to be discussed, of course.

[apognu]

Let's merge and improve afterwards.

[Pascal-Delange]

👍