GreptimeDb Prometheus benchmark

[mladjan-gadzic]

Hello community! We at G-Research are considering GreptimeDb as store for our immense observability data. With that in mind we'd like to run Prometheus benchmark as industry standard. This article https://greptime.com/blogs/2024-08-02-datanode-benchmark got our attention and we're looking to reproduce it.

Once everything is configured and Prometheus benchmark started, only single datanode is utilized. We think it has to do with how regions/partitions are being created.

Our benchmark setup is based on GreptimeDb on Kubernetes from here https://docs.greptime.com/user-guide/deployments/deploy-on-kubernetes/getting-started/. We used everything as described in docs but just on a bigger scale (3 datanodes, 3 frontends for the beginning). We're using vanilla configuration of Prometheus benchmark with changed writeUrl.

In autoconfigured Grafana dashboard we can see that all frontends are utilized which means that traffic is properly balanced but it seems that only one datanode is utilized. We are not sure why and how to fix this.

Any ideas?

Appreciate any feedback!

[sunng87]

We talked this in community slack, but in case other open sources users are looking at this, to spread prometheus workload across datanodes, you need to create physical table manually as describe in:

https://docs.greptime.com/user-guide/ingest-data/for-observability/prometheus#performance-tuning

• You don't need to specify all label columns in the physical table because they will be added automatically
• You need specify partition columns, because there is no good default prometheus labels for partitioning, it depends on your data pattern. For example, namespace from kubernetes might be a good candidate for that
• For performance consideration, you will also need to add index on certain columns. But this can be done later with alter table sql.

For more information about partitioning, see https://docs.greptime.com/user-guide/deployments-administration/manage-data/table-sharding/#partition

To improve this, we have been working on repartition feature, once it's done you won't need to do this type of partitioning ahead of time.

[mladjan-gadzic]

Thanks for the explanation! This cover more than discussed in Slack. I'll post here more info about progress made in case someone is interested in it.