Use oxide-tokio-rt rather than #[tokio::main]

[hawkw]

As described in RFD 579, we are now recommending certain non-default Tokio runtime configurations in all software we deploy in production. The oxide-tokio-rt crate provides a way to easily get these common recommended configurations by using its functions to initialize the runtime rather than #[tokio::main]

Presently, oxide-tokio-rt does the following:

• Enables DTrace probes for Tokio runtime events using tokio-dtrace
• Disables the LIFO slot optimization, which was the root cause of omicron#8334

But, we anticipate that if there are additional Tokio configurations which we want to recommend in the future, we'll add them here as well.

This commit updates dendrite to initialize the Tokio runtime using oxide-tokio-rt. I've added a Clippy config for warning on uses of #[tokio::main], as well, to avoid forgetting to use oxide-tokio-rt if new binaries are added. All the configs we set currently in oxide-tokio-rt requires the tokio_unstable RUSTFLAGS cfg, but it looks like that's already being set in order to use tokio-conssole, so that shouldn't be a problem.

[Nieuwejaar]

For the most part dendrite's use of tokio and async is pretty basic. If it weren't required for dropshot, I'm not sure we would be using it at all.

One way in which we are possibly more complicated than other components is that we need to play nicely with the standard illumos multithreading mechanisms. The tofino SDE on which we are built uses threads and mutexes internally for all of its asynchrony. There are times we call into thread-land from tokio code, and there are times we use tokio channels from the SDE's thread-land to send notifications back to the main body of our code.

All of that is just to say that this should really get a workout on real hardware before being integrated, even though it look syntactically straightforward.

[hawkw]

All of that is just to say that this should really get a workout on real hardware before being integrated, even though it look syntactically straightforward.

Yeah, I think it's definitely worth testing the change the same way you'd exercise any other change, but I think it's unlikely to break anything. Most of the risk associated with picking up this stuff in other repos has actually just been build/dev environment complexity due to requiring tokio_unstable --- the build.rustflags setting in .cargo/config.toml doesn't compose with other ways of setting RUSTFLAGS which is unfortunate when people are setting the env var or doing other things. That's less of an issue here because there's already a .cargo/config.toml that's enabling tokio-unstable, and dendrite already won't build without it.

[hawkw]

(Sorry for the CI churn --- I couldn't get Dendrite to build locally due to some kind of libclang symbol versioning mess, so I thought I'd just let CI do it... 😅)