Skip to content

aperture: internally aggregate session statistics to reduce cardinality #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 22, 2025

Conversation

djkazic
Copy link
Contributor

@djkazic djkazic commented May 6, 2025

This PR adds aggregation and classification of sessions into {active, standby, inuse} categories and makes direct metrics exportable.

It is an attempt to lower the cardinality of the prior used read count metric. This should greatly reduce the load on Prometheus for both storage and querying.

However, there is a caveat with the code as-is:

  • Unbounded memory growth: while the prior approach essentially made it the prometheus server's problem, we now have to deal with this ourselves. Looking for feedback on an appropriate culling threshold. I wanted to first and foremost mimic what we were doing before so I did not implement culling yet.

@djkazic djkazic force-pushed the aggregated-session-stats branch from e43f56a to 9fc2af7 Compare May 6, 2025 15:32
@djkazic
Copy link
Contributor Author

djkazic commented May 6, 2025

Pushing a lint fix.

@djkazic djkazic changed the title aperture: internally aggregate session statistics to reduce cardinality [HOLD] aperture: internally aggregate session statistics to reduce cardinality May 7, 2025
@lightninglabs-deploy
Copy link
Collaborator

@djkazic, remember to re-request review from reviewers when ready

@djkazic djkazic force-pushed the aggregated-session-stats branch from 9fc2af7 to 7233c25 Compare May 19, 2025 16:40
@djkazic
Copy link
Contributor Author

djkazic commented May 19, 2025

Force pushed some changes around TTL and formatting / comments. PTAL @Roasbeef

@djkazic djkazic requested a review from Roasbeef May 19, 2025 16:40
Copy link
Member

@Roasbeef Roasbeef left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ❄️

Copy link
Member

@guggero guggero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this.
Were you able to test this with the LNC test page?
Would perhaps make sense to make sure the numbers will be what we expect. Maybe @ViktorTigerstrom has time to do a test run?

@djkazic djkazic changed the title [HOLD] aperture: internally aggregate session statistics to reduce cardinality aperture: internally aggregate session statistics to reduce cardinality May 21, 2025
@djkazic djkazic requested a review from guggero May 21, 2025 15:29
Copy link
Member

@guggero guggero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK, LGTM 🎉

@bhandras bhandras merged commit 20564d4 into lightninglabs:master May 22, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants