Should we enable optimizations for dev builds by default?

[alexpyattaev]

See #5155 for context.
TL;DR of it is:

Pros:

• UB gets exposed in tests rather than in prod
• CPU bound tests become ~4x faster
• Compile times stay about the same
• We use less 4x less compute time to run test suites

Cons:

• Debugging via gdb may become less fun
• IO bound tests do not get any measurable benefits
• CI times do not actually improve much since CI is constrained by the slowest tests (which are typically IO bound and/or just run a cluster for 10 min)
• Unless we go with -O3 we are still getting different compiler behavior between tests and prod

A compromise may be to enable optimizer in CI only, which has the following features:

• UB gets exposed in CI rather than in prod
• Dev builds stay unchanged
• Figuring out why test flakes in CI becomes even more "fun" since we have different settings in CI and in dev

[willhickey]

UB gets exposed in CI rather than in prod

There's still no guarantee of exposing UB, right? Just a chance?

[alexpyattaev]

Even miri does not guarantee detection of all UB.
But there is a 100% guarantee of not exposing optimizer-induced UB if running without optimizations enabled =).

[ilya-bobyr]

A compromise may be to enable optimizer in CI only, which has the following features:

UB gets exposed in CI rather than in prod
Dev builds stay unchanged
Figuring out why test flakes in CI becomes even more "fun" since we have different settings in CI and in dev

I think that having a dedicated profile for CI with different optimization
settings is a better choice. So I feel calling it a "compromise" is inaccurate.
I think it is a good alternative.

And if we try to detect the undefined behavior, I think we actually have much
more chances of finding anything if we run tests with different levels of
optimization, rather than with any one specific level.

I agree that it is more complex to create and manage a separate profile.
But running a unit test with a different profile is a simple as adding a
--profile argument. Going from

./cargo test --package solana-core -- --nocapture replay_stage

to

./cargo test --profile ci --package solana-core -- --nocapture replay_stage

You do need to know that this is how you do it. But if our CI uses the profile
argument explicitly, then you would just copy/paste the command from the CI
output. Which is what I do when I need to debug a CI test failure anyway.

I also think that using O3 in this case could be an option that we may want to
consider. Which is not practical for local builds. So, this is another thing
to consider, when we think about a dedicated CI profile.

[alexpyattaev]

-O3 is a lot slower to build. CI will be slower if we do it.

[ilya-bobyr]

But this way, it would be running code that is more like the one we run in production.
We would probably want to keep the asserts and debug checks, so it still would not be 100% identical.

There are ways to speed up the compilation that we are not using, as far as I know.
So, a separate profile opens this up as a possibility.
If the slowdown is considerable, we can stay at O1, until we figure out a way to speed up the compilation though caching and parallelization to compensate.

[alexpyattaev]

@yihau @willhickey do we actually have to recompile everything for every buildkite test or can we build once and re-use artifacts when running tests?

[yihau]

Yeah, We can definitely have a build and upload step before fanning out the tests. it should save some time. However, I think the build time might not be the bottleneck until we enable the optimization and improve the local-cluster test.

[alexpyattaev]

build time is like 10% of the CI times (about 2 min out of 20 for a successful run). Also it may have to become longer due to #5316

[KirillLykov]

CI times do not actually improve much since CI is constrained by the slowest tests (which are typically IO bound and/or just run a cluster for 10 min)

Could you provide some numbers for that? I honestly thought that the whole point is to speed up CI.

[alexpyattaev]

In #5155 there are some numbers based on local measurements on a devbox. TL;DR is that time to run tests is bottlenecked by the slowest test in a batch, not how fast average test runs, and slowest tests are not compute bound. Real CI timings are all over the place, just re-running it can change values by minutes.