Building CMLFS with LTO

[dslm4515]

LTO, Link-Time Optimization, promises faster execution time in exchange for longer compile time and possible smaller binaries/libraries. There is also ThinLTO that is like LTO but with short compile times than 'Full-LTO'

Binutils, LLVM, and GCC of llvmtools would need LTO feature turned on.

For GCC, add to configure command:

--enable-stage1-languages=c,lto 
--enable-languages=c,c++,lto
--with-build-config=bootstrap-lto
--enable-lto

For LLVM, add to cmake command:

-DLLVM_ENABLE_LTO=ON

Set flags to use LTO:

CFLAGS="-flto"
CXXFLAGS="-flto"
LDFLAGS="-flto -Wl,-O2 -Wl,--as-needed"
# When adding additional flags, -flto has to be first. 

or for ThinLTO:

CFLAGS="-flto=thin"
CXXFLAGS="-flto=thin"
LDFLAGS="-flto -Wl,-O2 -Wl,--as-needed" 
# When adding additional flags, -flto has to be first.

Per owl4ce (for more optimization):

You may want to use -march=<arch> (prefer with native) as well. 
Also I recommend to use -pipe flag to C and CXX flags.

For LTO and more optimization:

export CFLAGS="-flto -pipe -Oz -fno-semantic-interposition "
export CXXFLAGS="-flto -pipe -Oz -fno-semantic-interposition "
export LDFLAGS="-flto -Wl,-O2 -Wl,--as-needed"
# Use -Oz only with clang, as GCC does not recognize it

Interesting projects:

https://github.com/InBetweenNames/gentooLTO

[dslm4515]

For build-time comparison:

# Building LLVM-12 under chroot with 2 job threads (`make -j2`)
non-LTO: 171m27.003s
LTO-built: 218m57.358s

Host machine specs: AMD Phenom(tm) II X2 550 (2 x 3.1 GHz), 4GB DDR2 800, SATA II HDD

[dslm4515]

LTO Status:

cgnutools:

• binutils, builds fine with --enable-lto ... 7min27.310s
• gcc-static - builds fine (with above flags) ... 10m27.899s Do not enable LTO
• musl libc - will not compile with LTO (unresolved symbol) ... 2m10.369s
• gcc-final - builds fine (with above flags) ... 26m9.678s Do not enable LTO
• file - builds fine (with above flags) ... 8.001s
• libexecinfo - builds fine (with above flags)
• ncurses - have not tried LTO
• LLVM - build fine with -DLLVM_ENABLE_LTO=ON ... 171m5958s

llvmtools:

• Musl libc - will not compile with LTO (unresolved symbol)
• libunwind - builds fine (with above flags) ... 7.86s
• libcxxabi - builds fine (with above flags) ... 7.852s
• libcxx - builds fine (with above flags) ... 54.155s
• libexecinfo - builds fine (with above flags) ... 2.592s
• ncurses - builds fine (with above flags) ... 24.203s
• LLVM - pending, so far compiling for 24 hours and counting!

[dslm4515]

Update: Do not build GCC with LTO for cgnutools.

Seems to cause issues with file failing to configure unless --host is defined and cgnutool's GCC unable to find libcurses when building LLVM. Also file segfaults if GCC of cgnutools was built with LTO.

[dslm4515]

Update: Ncurses required as a shared library in cgnutools

Build fails to link with.. static ncurses?

[ 54%] Linking CXX shared library ../../lib/libLTO.so                                                                                           
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cg
nutools/lib/../lib/libcurses.a(lib_cur_term.o): relocation R_X86_64_32 against symbol `ttytype' can not be used when making a shared object; rec
ompile with -fPIC                                                                                                                               
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cg
nutools/lib/../lib/libcurses.a(lib_setup.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recomp
ile with -fPIC                                                                                                                                  
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(lib_tputs.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(read_entry.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(alloc_ttype.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(comp_error.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(comp_userdefs.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile
 with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(db_iterator.o): relocation R_X86_64_32S against symbol `_nc_globals' can not be used when making a shared object;
 recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(getenv_num.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(home_terminfo.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(lib_baudrate.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile
 with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cg
nutools/lib/../lib/libcurses.a(lib_options.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(comp_captab.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(init_keytry.o): relocation R_X86_64_32 against symbol `_nc_tinfo_fkeys' can not be used when making a shared object; recompile with -fPIC
/mnt/cmlto/cgnutools/bin/../lib/gcc/x86_64-cmlfs-linux-musl/10.3.1/../../../../x86_64-cmlfs-linux-musl/bin/ld: /mnt/cmlto/cgnutools/bin/../../cgnutools/lib/../lib/libcurses.a(lib_tputs.o): warning: relocation against `stdout' in read-only section `.text'
collect2: error: ld returned 1 exit status
make[2]: *** [tools/lto/CMakeFiles/LTO.dir/build.make:166: lib/libLTO.so.12] Error 1
make[2]: Leaving directory '/mnt/cmlto/sources/llvm/build'
make[1]: *** [CMakeFiles/Makefile2:15804: tools/lto/CMakeFiles/LTO.dir/all] Error 2
make[1]: Leaving directory '/mnt/cmlto/sources/llvm/build'
make: *** [Makefile:156: all] Error 2
make: Leaving directory '/mnt/cmlto/sources/llvm/build'

[dslm4515]

Issue fixed. Ncurses should not be static, must be built as a shared library

[dslm4515]

I do wonder if stripping (--strip-unneeded and --strip-debug) is necessary or optional. Not sure if stripping is an indication if package was successfully compiled with LTO:

# when stripping binaries and libraries of bison:
 /llvmtools/bin/strip: /BUILD/usr/lib/liby.a(main.o): Unable to recognise the format of file: file format not recognized
/llvmtools/bin/strip: /BUILD/usr/lib/liby.a(yyerror.o): Unable to recognise the format of file: file format not recognized

Or is the above warning for static libraries built with LTO?

[dslm4515]

Looks like if binaries & libraries are stripped with strip and the above error appears (file format *.o is unrecognized) then building that package with LTO failed.

[dslm4515]

List of 'LTO' status of packages:

• Python 3.9.5 - no -flto (configure finds an error), no --enable-optimizations since system LLVM is missing libclang_rt.profile-x86_64.a
• argp-standalone - no -flto (builds but libraries are useless)
• zlib/zlib-ng - no -flto (builds but libraries are useless)
• e2fsprogs - no -flto (builds but libraries are useless)
• execline, skalibg, utmps, s6, s6-rc, s6 suite - no -flto (builds but libraries are useless)

[owl4ce]

About missing libclang_rt.profile-x86_64.a

[owl4ce]

Untested, I'm not sure ..

-DLLVM_ENABLE_PROJECTS
https://clang.llvm.org/get_started.html

[owl4ce]

I confirm, need to enable profiler or something (disabled in this repo).

Gentoo has separated package to enable sanitizer.
https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/compiler-rt-sanitizers/compiler-rt-sanitizers-12.0.1.ebuild

[owl4ce]

I suggest to use -Wno-dev while configure LLVM to suppress developer warnings.

[dslm4515]

I will likely open an new repo with optimization in mind and CMLFS as the base

[dslm4515]

I may call it ... Optimux-C

[ghost]

I was just going to try and build optimux with lld since I've heard that it's faster than gold.

I guess optimux-c will be a frankenstein of all of your repos.

[dslm4515]

With my 2nd build of optimux, I will try to use the gold linker... since its faster than ld.bfd.

But yes, with Optimux-C, i too will build it with LLD

[dslm4515]

Quick research shows GCC only has 1-3 % performance gain (when optimizing for execution speed primarily and size secondarily) over LLVM built libraries and binaries.

I suppose would not recommend Optimux-C for embedded systems due to a potentially larger rootfs compared to a GCC built rootfs.