Speed up incremental compilation time of tests

[Scony]

Problem

At the moment, in current master (as of 11th February 2023) the incremental compilation of tests takes a very long time. In my case it's ~2 minutes after a tiny change in 1 test case.

Example

What I mean by incremental is as follows - suppose I've just built an engine with tests using the command:

scons -j$(nproc) tests=yes debug_symbols=yes optimize=debug verbose=yes CXX='time g++' LINK='time g++'

now, if I change one line of code in any of the tests which are then included by tests/test_main.cpp, and run the above command again I'll get this:

$ scons -j$(nproc) tests=yes debug_symbols=yes optimize=debug verbose=yes CXX='time g++' LINK='time g++'
scons: Reading SConscript files ...
Automatically detected platform: linuxbsd
Building for platform "linuxbsd", architecture "x86_64", target "editor".
Couldn't parse CXX environment variable to infer compiler version.
Couldn't detect compiler version, skipping version checks. Build may fail if the compiler doesn't support C++17 fully.
Checking for C header file mntent.h... (cached) yes
scons: done reading SConscript files.
scons: Building targets ...
[  8%] time g++ -o tests/test_main.linuxbsd.editor.x86_64.o -c -std=gnu++17 -pipe -gdwarf-4 -g2 -Og -Wall -Wshadow-local -Wno-misleading-indentation -Wno-type-limits -DTOOLS_ENABLED -DDEBUG_ENABLED -DNDEBUG -DNO_EDITOR_SPLASH -DTOUCH_ENABLED -DFONTCONFIG_ENABLED -DALSA_ENABLED -DALSAMIDI_ENABLED -DPULSEAUDIO_ENABLED -D_REENTRANT -DDBUS_ENABLED -DSPEECHD_ENABLED -DJOYDEV_ENABLED -DUDEV_ENABLED -DLINUXBSD_ENABLED -DUNIX_ENABLED -D_FILE_OFFSET_BITS=64 -DX11_ENABLED -DVULKAN_ENABLED -DGLES3_ENABLED -DMINIZIP_ENABLED -DZSTD_STATIC_LINKING_ONLY -DUSE_VOLK -DVK_USE_PLATFORM_XLIB_KHR -DGLAD_ENABLED -DGLES_OVER_GL -Ithirdparty/freetype/include -Ithirdparty/libpng -Ithirdparty/glad -Ithirdparty/volk -Ithirdparty/vulkan -Ithirdparty/vulkan/include -Ithirdparty/zstd -Ithirdparty/zlib -Iplatform/linuxbsd -Ithirdparty/linuxbsd_headers -I. tests/test_main.cpp
[ 99%] progress_finish(["progress_finish"], [])
[100%] 
real	1m29.215s
user	1m31.197s
sys	0m1.710s
[100%] time g++ -o bin/godot.linuxbsd.editor.x86_64 -static-libgcc -static-libstdc++ platform/linuxbsd/godot_linuxbsd.linuxbsd.editor.x86_64.o platform/linuxbsd/crash_handler_linuxbsd.linuxbsd.editor.x86_64.o platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o platform/linuxbsd/joypad_linux.linuxbsd.editor.x86_64.o platform/linuxbsd/freedesktop_portal_desktop.linuxbsd.editor.x86_64.o platform/linuxbsd/freedesktop_screensaver.linuxbsd.editor.x86_64.o platform/linuxbsd/xkbcommon-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/display_server_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/key_mapping_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xlib-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xcursor-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xinerama-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xinput2-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xrandr-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xrender-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xext-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/vulkan_context_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/gl_manager_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/detect_prime_x11.linuxbsd.editor.x86_64.o thirdparty/glad/glx.linuxbsd.editor.x86_64.o platform/linuxbsd/speechd-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/tts_linux.linuxbsd.editor.x86_64.o platform/linuxbsd/fontconfig-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/libudev-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/dbus-so_wrap.linuxbsd.editor.x86_64.o tests/test_macros.linuxbsd.editor.x86_64.o tests/test_main.linuxbsd.editor.x86_64.o tests/test_navigation_main.linuxbsd.editor.x86_64.o tests/test_utils.linuxbsd.editor.x86_64.o main/libmain.linuxbsd.editor.x86_64.a modules/libmodules.linuxbsd.editor.x86_64.a modules/libmodule_msdfgen.linuxbsd.editor.x86_64.a modules/libmodule_theora.linuxbsd.editor.x86_64.a modules/libmodule_vorbis.linuxbsd.editor.x86_64.a modules/libmodule_astcenc.linuxbsd.editor.x86_64.a modules/libmodule_basis_universal.linuxbsd.editor.x86_64.a modules/libmodule_bmp.linuxbsd.editor.x86_64.a modules/libmodule_csg.linuxbsd.editor.x86_64.a modules/libmodule_cvtt.linuxbsd.editor.x86_64.a modules/libmodule_dds.linuxbsd.editor.x86_64.a modules/libmodule_denoise.linuxbsd.editor.x86_64.a modules/libmodule_enet.linuxbsd.editor.x86_64.a modules/libmodule_etcpak.linuxbsd.editor.x86_64.a modules/libmodule_freetype.linuxbsd.editor.x86_64.a modules/libmodule_gdscript.linuxbsd.editor.x86_64.a modules/libmodule_glslang.linuxbsd.editor.x86_64.a modules/libmodule_gltf.linuxbsd.editor.x86_64.a modules/libmodule_gridmap.linuxbsd.editor.x86_64.a modules/libmodule_hdr.linuxbsd.editor.x86_64.a modules/libmodule_jpg.linuxbsd.editor.x86_64.a modules/libmodule_jsonrpc.linuxbsd.editor.x86_64.a modules/libmodule_lightmapper_rd.linuxbsd.editor.x86_64.a modules/libmodule_mbedtls.linuxbsd.editor.x86_64.a modules/libmodule_meshoptimizer.linuxbsd.editor.x86_64.a modules/libmodule_minimp3.linuxbsd.editor.x86_64.a modules/libmodule_mobile_vr.linuxbsd.editor.x86_64.a modules/libmodule_multiplayer.linuxbsd.editor.x86_64.a modules/libmodule_navigation.linuxbsd.editor.x86_64.a modules/libmodule_noise.linuxbsd.editor.x86_64.a modules/libmodule_ogg.linuxbsd.editor.x86_64.a modules/libmodule_openxr.linuxbsd.editor.x86_64.a modules/libmodule_raycast.linuxbsd.editor.x86_64.a modules/libmodule_regex.linuxbsd.editor.x86_64.a modules/libmodule_squish.linuxbsd.editor.x86_64.a modules/libmodule_svg.linuxbsd.editor.x86_64.a modules/libmodule_text_server_adv.linuxbsd.editor.x86_64.a modules/libmodule_tga.linuxbsd.editor.x86_64.a modules/libmodule_tinyexr.linuxbsd.editor.x86_64.a modules/libmodule_upnp.linuxbsd.editor.x86_64.a modules/libmodule_vhacd.linuxbsd.editor.x86_64.a modules/libmodule_webp.linuxbsd.editor.x86_64.a modules/libmodule_webrtc.linuxbsd.editor.x86_64.a modules/libmodule_websocket.linuxbsd.editor.x86_64.a modules/libmodule_webxr.linuxbsd.editor.x86_64.a modules/libmodule_xatlas_unwrap.linuxbsd.editor.x86_64.a modules/libmodule_zip.linuxbsd.editor.x86_64.a platform/libplatform.linuxbsd.editor.x86_64.a drivers/libdrivers.linuxbsd.editor.x86_64.a editor/libeditor.linuxbsd.editor.x86_64.a scene/libscene.linuxbsd.editor.x86_64.a servers/libservers.linuxbsd.editor.x86_64.a core/libcore.linuxbsd.editor.x86_64.a modules/text_server_adv/libharfbuzz_builtin.linuxbsd.editor.x86_64.a modules/text_server_adv/libgraphite_builtin.linuxbsd.editor.x86_64.a modules/text_server_adv/libicu_builtin.linuxbsd.editor.x86_64.a modules/freetype/libfreetype_builtin.linuxbsd.editor.x86_64.a modules/msdfgen/libmsdfgen_builtin.linuxbsd.editor.x86_64.a -lpthread -ldl

real	0m18.403s
user	0m15.777s
sys	0m2.479s
[100%] scons: done building targets.
[Time elapsed: 00:01:54.996]

which reads as follows:

• I had to wait ~1m 55s for the build to complete so that I can run tests
• compilation of that single, big translation unit which contains all the test cases took 1m29.215s
• linking of the executable took 0m18.403s

Looking at the above the issue seems quite obvious - the compilation is the major problem.

Solution

How can one improve the overall incremental build time in that case? There are at least 3 options:

1. use faster build tools like different compiler/linker etc.
2. split single, big translation unit into pieces and compile them separately - likely in parallel
3. remove some tests

Option 3 is not an option obviously. Option 1 looks quite easy, but in reality, it does not scale with the number of test cases. Option 2, however, is not that complex and it scales perfectly well with the number of test cases i.e. if some translation unit becomes big, it can be split into pieces.

Example again

Suppose we've decided to introduce a new test case but as a separate translation unit - just like in this PR: godotengine/godot#73108. Moreover, suppose we've already built the tests and run them.

Now, if we change one line in the test_navigation_server_3d.h and run:

scons -j$(nproc) tests=yes debug_symbols=yes optimize=debug verbose=yes CXX='time g++' LINK='time g++'

we will get:

$ scons -j$(nproc) tests=yes debug_symbols=yes optimize=debug verbose=yes CXX='time g++' LINK='time g++'
scons: Reading SConscript files ...
Automatically detected platform: linuxbsd
Building for platform "linuxbsd", architecture "x86_64", target "editor".
Couldn't parse CXX environment variable to infer compiler version.
Couldn't detect compiler version, skipping version checks. Build may fail if the compiler doesn't support C++17 fully.
Checking for C header file mntent.h... (cached) yes
scons: done reading SConscript files.
scons: Building targets ...
[  8%] time g++ -o tests/test_navigation_main.linuxbsd.editor.x86_64.o -c -std=gnu++17 -pipe -gdwarf-4 -g2 -Og -Wall -Wshadow-local -Wno-misleading-indentation -Wno-type-limits -DTOOLS_ENABLED -DDEBUG_ENABLED -DNDEBUG -DNO_EDITOR_SPLASH -DTOUCH_ENABLED -DFONTCONFIG_ENABLED -DALSA_ENABLED -DALSAMIDI_ENABLED -DPULSEAUDIO_ENABLED -D_REENTRANT -DDBUS_ENABLED -DSPEECHD_ENABLED -DJOYDEV_ENABLED -DUDEV_ENABLED -DLINUXBSD_ENABLED -DUNIX_ENABLED -D_FILE_OFFSET_BITS=64 -DX11_ENABLED -DVULKAN_ENABLED -DGLES3_ENABLED -DMINIZIP_ENABLED -DZSTD_STATIC_LINKING_ONLY -DUSE_VOLK -DVK_USE_PLATFORM_XLIB_KHR -DGLAD_ENABLED -DGLES_OVER_GL -Ithirdparty/freetype/include -Ithirdparty/libpng -Ithirdparty/glad -Ithirdparty/volk -Ithirdparty/vulkan -Ithirdparty/vulkan/include -Ithirdparty/zstd -Ithirdparty/zlib -Iplatform/linuxbsd -Ithirdparty/linuxbsd_headers -I. tests/test_navigation_main.cpp
[ 25%] 
real	0m1.978s
user	0m1.874s
sys	0m0.121s
[100%] progress_finish(["progress_finish"], [])
[100%] time g++ -o bin/godot.linuxbsd.editor.x86_64 -static-libgcc -static-libstdc++ platform/linuxbsd/godot_linuxbsd.linuxbsd.editor.x86_64.o platform/linuxbsd/crash_handler_linuxbsd.linuxbsd.editor.x86_64.o platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o platform/linuxbsd/joypad_linux.linuxbsd.editor.x86_64.o platform/linuxbsd/freedesktop_portal_desktop.linuxbsd.editor.x86_64.o platform/linuxbsd/freedesktop_screensaver.linuxbsd.editor.x86_64.o platform/linuxbsd/xkbcommon-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/display_server_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/key_mapping_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xlib-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xcursor-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xinerama-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xinput2-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xrandr-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xrender-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/dynwrappers/xext-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/vulkan_context_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/gl_manager_x11.linuxbsd.editor.x86_64.o platform/linuxbsd/x11/detect_prime_x11.linuxbsd.editor.x86_64.o thirdparty/glad/glx.linuxbsd.editor.x86_64.o platform/linuxbsd/speechd-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/tts_linux.linuxbsd.editor.x86_64.o platform/linuxbsd/fontconfig-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/libudev-so_wrap.linuxbsd.editor.x86_64.o platform/linuxbsd/dbus-so_wrap.linuxbsd.editor.x86_64.o tests/test_macros.linuxbsd.editor.x86_64.o tests/test_main.linuxbsd.editor.x86_64.o tests/test_navigation_main.linuxbsd.editor.x86_64.o tests/test_utils.linuxbsd.editor.x86_64.o main/libmain.linuxbsd.editor.x86_64.a modules/libmodules.linuxbsd.editor.x86_64.a modules/libmodule_msdfgen.linuxbsd.editor.x86_64.a modules/libmodule_theora.linuxbsd.editor.x86_64.a modules/libmodule_vorbis.linuxbsd.editor.x86_64.a modules/libmodule_astcenc.linuxbsd.editor.x86_64.a modules/libmodule_basis_universal.linuxbsd.editor.x86_64.a modules/libmodule_bmp.linuxbsd.editor.x86_64.a modules/libmodule_csg.linuxbsd.editor.x86_64.a modules/libmodule_cvtt.linuxbsd.editor.x86_64.a modules/libmodule_dds.linuxbsd.editor.x86_64.a modules/libmodule_denoise.linuxbsd.editor.x86_64.a modules/libmodule_enet.linuxbsd.editor.x86_64.a modules/libmodule_etcpak.linuxbsd.editor.x86_64.a modules/libmodule_freetype.linuxbsd.editor.x86_64.a modules/libmodule_gdscript.linuxbsd.editor.x86_64.a modules/libmodule_glslang.linuxbsd.editor.x86_64.a modules/libmodule_gltf.linuxbsd.editor.x86_64.a modules/libmodule_gridmap.linuxbsd.editor.x86_64.a modules/libmodule_hdr.linuxbsd.editor.x86_64.a modules/libmodule_jpg.linuxbsd.editor.x86_64.a modules/libmodule_jsonrpc.linuxbsd.editor.x86_64.a modules/libmodule_lightmapper_rd.linuxbsd.editor.x86_64.a modules/libmodule_mbedtls.linuxbsd.editor.x86_64.a modules/libmodule_meshoptimizer.linuxbsd.editor.x86_64.a modules/libmodule_minimp3.linuxbsd.editor.x86_64.a modules/libmodule_mobile_vr.linuxbsd.editor.x86_64.a modules/libmodule_multiplayer.linuxbsd.editor.x86_64.a modules/libmodule_navigation.linuxbsd.editor.x86_64.a modules/libmodule_noise.linuxbsd.editor.x86_64.a modules/libmodule_ogg.linuxbsd.editor.x86_64.a modules/libmodule_openxr.linuxbsd.editor.x86_64.a modules/libmodule_raycast.linuxbsd.editor.x86_64.a modules/libmodule_regex.linuxbsd.editor.x86_64.a modules/libmodule_squish.linuxbsd.editor.x86_64.a modules/libmodule_svg.linuxbsd.editor.x86_64.a modules/libmodule_text_server_adv.linuxbsd.editor.x86_64.a modules/libmodule_tga.linuxbsd.editor.x86_64.a modules/libmodule_tinyexr.linuxbsd.editor.x86_64.a modules/libmodule_upnp.linuxbsd.editor.x86_64.a modules/libmodule_vhacd.linuxbsd.editor.x86_64.a modules/libmodule_webp.linuxbsd.editor.x86_64.a modules/libmodule_webrtc.linuxbsd.editor.x86_64.a modules/libmodule_websocket.linuxbsd.editor.x86_64.a modules/libmodule_webxr.linuxbsd.editor.x86_64.a modules/libmodule_xatlas_unwrap.linuxbsd.editor.x86_64.a modules/libmodule_zip.linuxbsd.editor.x86_64.a platform/libplatform.linuxbsd.editor.x86_64.a drivers/libdrivers.linuxbsd.editor.x86_64.a editor/libeditor.linuxbsd.editor.x86_64.a scene/libscene.linuxbsd.editor.x86_64.a servers/libservers.linuxbsd.editor.x86_64.a core/libcore.linuxbsd.editor.x86_64.a modules/text_server_adv/libharfbuzz_builtin.linuxbsd.editor.x86_64.a modules/text_server_adv/libgraphite_builtin.linuxbsd.editor.x86_64.a modules/text_server_adv/libicu_builtin.linuxbsd.editor.x86_64.a modules/freetype/libfreetype_builtin.linuxbsd.editor.x86_64.a modules/msdfgen/libmsdfgen_builtin.linuxbsd.editor.x86_64.a -lpthread -ldl

real	0m18.401s
user	0m15.706s
sys	0m2.467s
[100%] scons: done building targets.
[Time elapsed: 00:00:37.332]

which reads as:

• I had to wait ~37s for the build to complete so that I can run tests
• compilation of the only affected translation unit (pretty small one) took 1.978s
• linking of the executable took 18.401s

Conclusions

incremental compilation of smaller translation unit - independent from others - resulted in build time of tests faster by order of magnitude.

This way, writing tests is much more efficient and more encouraging - especially to non-advanced C++ programmers.

Proposal

Granted the above, I propose to split tests into multiple translation units.