Improve memory use of rabbit_mgmt_gc
#13898
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `rabbit_mgmt_gc` gen_server performs garbage collections periodically. When doing so it can create potentially fairly large terms, for example by creating a set out of `rabbit_exchange:list_names/0`. With many exchanges, for example, the process memory usage can climb steadily especially when the management agent is mostly idle since `rabbit_mgmt_gc` won't hit enough reductions to cause a full-sweep GC on itself. Since the process is only active periodically (once every 2min by default) we can hibernate it to GC the terms it created. This can save a medium amount of memory in situations where there are very many pieces of metadata (exchanges, vhosts, queues, etc.). For example on an idle single-node broker with 50k exchanges, `rabbit_mgmt_gc` can hover around 50MB before being naturally GC'd. With this patch the process memory usage stays consistent between `start_gc` timer messages at around 1KB.
`sets` v2 were not yet available when this module was written. Compared
to `gb_sets`, v2 `sets` are faster and more memory efficient:
> List = lists:seq(1, 50_000).
> tprof:profile(sets, from_list, [List, [{version, 2}]], #{type => call_memory}).
****** Process <0.94.0> -- 100.00% of total ***
FUNCTION CALLS WORDS PER CALL [ %]
maps:from_keys/2 1 184335 184335.00 [100.00]
184335 [ 100.0]
ok
> tprof:profile(gb_sets, from_list, [List], #{type => call_memory}).
****** Process <0.97.0> -- 100.00% of total ***
FUNCTION CALLS WORDS PER CALL [ %]
lists:rumergel/3 1 2 2.00 [ 0.00]
gb_sets:from_ordset/1 1 3 3.00 [ 0.00]
lists:reverse/2 1 100000 100000.00 [16.76]
lists:usplit_1/5 49999 100002 2.00 [16.76]
gb_sets:balance_list_1/2 65535 396605 6.05 [66.48]
596612 [100.0]
michaelklishin
approved these changes
May 16, 2025
michaelklishin
added a commit
that referenced
this pull request
May 17, 2025
Improve memory use of `rabbit_mgmt_gc` (backport #13898)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The main change here is to hibernate the
rabbit_mgmt_gcgen_server after it completes its GC run. It's an ideal process for hibernation since it wakes up periodically (every 2min by default) to do some work and is then completely idle.Especially if the broker is mostly idle this server may not perform enough work to be GC'd naturally so its process memory use can grow steadily over time. It can get up to fairly high amounts (tens of MB) because of the work it does during each GC run: it creates a set out of metadata entities like vhosts, queues and exchanges. On an idle single-node broker with 50k exchanges for example,
rabbit_mgmt_gccan creep up to around 50MB. With the hibernate change it stays at around 1KB between GC runs.I've also updated the sets usage here from
gb_setstosetsv2 as it's faster and more memory efficient.tprofcomparison of sets...On
mainyou can monitorrabbit_mgmt_gcwithobserver_cliorreconand create or import 50K exchanges. It will eventually creep up in MB of memory usage. Decrease the default time between wake-ups (rabbit_mgmt_gc.erl:23) to see it happen faster.