How do we want to expose configuration of thread pool sizes in agave?

[t-nelson]

Agave spawns an absurd number of threads. Far more than execution units on the CPU. There's been an ongoing, but disorganized effort to rein this in. Let's get on the same page before moving forward

[t-nelson]

IMO we need no more than CLI args for this

Can someone who's looked at it more recently give an overview of the current pools and their current sizes?

[steviez]

I had been using Thread and Thread Pool Audit to track "reduce thread count" work.

As part of this, I had a google sheet that I was using for my own sake; I'll update that with latest and post the link when I have that.

[alexpyattaev]

We have so many thread pools that configuring them all with CLI is very daunting. I'd go with a config file + sane defaults.

[steviez]

configuring them all with CLI is very daunting ... sane defaults.

Sane default is a must either way. My approach has been wiring up CLI arg with existing default, and then shrinking from there. And sane default means there should not be need for validators to tune the values in most cases. Adventurous operators can tweak things if they like, but the goal is that we'll tune values and update default values once we have better values. And the ability to change value without rebuilding the bin is our emergency option (hopefully never necessary if our testing is thorough enough)

I'd go with a config file

The config file would presumably be read somewhere higher up (ie Validator), and then values would need to be plumbed to their respective services. Plumbing values to each service needs to happen regardless of config file or CLI flags. So, the only different code should be parsing CLI args or parsing config file.

As for config file, I agree this might be a good longer term approach. However, I think we can make more immediate progress with CLI args. CLI args are what we currently use to configure everything else in the validator, so using more CLI args is consistent with our current approach. The alternatives are either porting all CLI config to config file OR living in some hybrid state where some stuff is configured with CLI / some stuff configured with config file (this doesn't seem great)

Also, Yihau is working on getting us towards a newer version of CLAP in validator (by first doing a bunch of cleanup & adding tests so we can feel confident that the upgrade is valid). The newer version of CLAP supports parsing arguments from ENV variables if I recall correctly. So, we'd get support for config file (via ENV variables in config file) OR CLI with the upgrade for free. Thus, I think we should forego the introduction of a config file just for these threadpools.

The actual performance gains can be hard regardless of whether we use CLI or config file, so I think we should stick with status quo of CLI

[bw-solana]

Well said by Steve.

The easy part we can agree on is the sane defaults w/ optionality and plumbing this down to all the thread pool creations.

As far as how we expose the configurability, I personally prefer a master config file (e.g. a TOML like FD uses). But I would rather just use command line than a hybrid cmd line for most + config file for threads.

[alexpyattaev]

I agree that config format is secondary to the key problem - we have too many runtimes. As long as each crate spawns its own rayon pool with 32 threads in it, we will keep having this problem, CLI args or otherwise. We need to be able to systematically supply runtimes from top to bottom, rather than spawning them inside individual crates. This way we can eventually get the number of threads to some value that lines up with the number of CPU cores.

Second issue is that we currently have no way to control core affinity and priority of the threads we make , so we end up with sporadic latency spikes all over the place. For example, we can have sigverify providing more shreds than retransmit stage can digest, which results in massive latency spikes. Giving higher priority to retransmit could help with that.

As long as we keep spawning runtimes at will, these problems are not going away.

[bw-solana]

Maybe we can use @steviez 's spreadsheet as a starting point and theorize which pools could be combined

[alexpyattaev]

I think there are 3 mostly independent efforts necessary:

• Making changes to patch thread pools from the upper layers (struct Validator or similar) down to crates instead of lazy_static all over the place (so they can get merged)
• Deciding which pools can be candidates and trying out options for merging
• Enabling customization of the thread pools as appropriate (for things like priority and affinity)

Thread manager tried to be all 3 and that was obviously too wide of a scope.