5.12.0 various issues with patchset 5.12-3, GCC 11.1 on Ubuntu 21.04

[ms178]

I am on Ubuntu 21.04 with the GCC 11.1-v3 toolchain (gcc version 11.1.0 (Ubuntu 11.1.0-1ubuntu1~21.04+v3).

During the compilation I get the following issues during the compilation stage:

LTO vmlinux.o
In function ‘memcmp’,
inlined from ‘tboot_probe’ at arch/x86/kernel/tboot.c:70:6:
./include/linux/fortify-string.h:235:16: warning: ‘__builtin_memcmp_eq’ specified bound 16 exceeds source size 0 [-Wstringop-overread]
235 | return __underlying_memcmp(p, q, size);
| ^
arch/x86/mm/pgtable.c: In function ‘pgd_alloc’:
arch/x86/mm/pgtable.c:437:13: warning: ‘preallocate_pmds.constprop’ accessing 8 bytes in a region of size 0 [-Wstringop-overflow=]
437 | if (preallocate_pmds(mm, pmds, PREALLOCATED_PMDS) != 0)
| ^
arch/x86/mm/pgtable.c:437:13: note: referencing argument 2 of type ‘struct pmd_t * *’
arch/x86/mm/pgtable.c:225:12: note: in a call to function ‘preallocate_pmds.constprop’
225 | static int preallocate_pmds(struct mm_struct *mm, pmd_t *pmds[], int count)
| ^
arch/x86/mm/pgtable.c:440:13: warning: ‘preallocate_pmds.constprop’ accessing 8 bytes in a region of size 0 [-Wstringop-overflow=]
440 | if (preallocate_pmds(mm, u_pmds, PREALLOCATED_USER_PMDS) != 0)
| ^
arch/x86/mm/pgtable.c:440:13: note: referencing argument 2 of type ‘struct pmd_t * *’
arch/x86/mm/pgtable.c:225:12: note: in a call to function ‘preallocate_pmds.constprop’
225 | static int preallocate_pmds(struct mm_struct *mm, pmd_t *pmds[], int count)
| ^
arch/x86/mm/pgtable.c:464:9: warning: ‘free_pmds.constprop’ accessing 8 bytes in a region of size 0 [-Wstringop-overflow=]
464 | free_pmds(mm, pmds, PREALLOCATED_PMDS);
| ^
arch/x86/mm/pgtable.c:464:9: note: referencing argument 2 of type ‘struct pmd_t * *’
arch/x86/mm/pgtable.c:213:13: note: in a call to function ‘free_pmds.constprop’
213 | static void free_pmds(struct mm_struct *mm, pmd_t *pmds[], int count)
| ^
OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: unexpected relocation symbol type in .rela.discard.func_stack_frame_non_standard: 0
vmlinux.o: warning: objtool: crc_pcl()+0x87: sibling call from callable instruction with modified stack frame
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo
GEN modules.builtin

Plus tons of these:

lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
LTO [M] net/netfilter/ipvs/ip_vs_ovf.lto.o
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
LTO [M] net/netfilter/ipvs/ip_vs_pe_sip.lto.o
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
LTO [M] net/netfilter/ipvs/ip_vs_rr.lto.o
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
LTO [M] net/netfilter/ipvs/ip_vs_sed.lto.o
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
LTO [M] net/netfilter/ipvs/ip_vs_sh.lto.o
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
LTO [M] net/netfilter/ipvs/ip_vs_wlc.lto.o
LTO [M] net/netfilter/ipvs/ip_vs_wrr.lto.o

I've used the vanilla sources from kernel.org. Here are the used config and makefile:

config.txt
Makefile.txt

[ms178]

P.S: The kernel also doesn't boot successfully, it hangs/crashes right after GRUB while loading from the Ramdisk and I have to reset the computer and boot my old Kernel to get the system to work again.

I could also reproduce the problems while using the Xanmod sources, and only using -O2 and -march=haswell as compilation flags.

[ms178]

The preallocate_pmds.constprop related warnings also appear in this GCC bug report: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99578

[ms178]

The Kernel still doesn't build with the updated 5.12-3 patchset from the following comment (InBetweenNames/gentooLTO#90 (comment)). While playing around with the Xanmod sources and this newer patchset, I've also come across various ICEs of GCC-11.1 which I have documented here (InBetweenNames/gentooLTO#90 (comment)) and here (InBetweenNames/gentooLTO#90 (comment)).

[andikleen]

Can you provide kernel .config files for the compiler crashes please?

[ms178]

@andikleen

I've just triggered another ICE:

CC [M] net/netfilter/xt_rateest.o
net/6lowpan/nhc_fragment.c:23:1: internal compiler error: Speicherzugriffsfehler [memory access fault]
23 | MODULE_LICENSE("GPL");
| ^~~~~~~~~~~~~~
CC [M] drivers/rtc/rtc-rp5c01.o
CC [M] drivers/edac/ie31200_edac.o
0xd8444f internal_error(char const*, ...)
???:0
CC [M] drivers/md/md-multipath.o
CC [M] fs/afs/fs_operation.o

Makefile (modified with custom compiler flags around line 417 pp. and line 470 pp.):
Makefile.txt
config:
config.txt

Some additional observations, I couldn't trigger ICEs with -O2 or -O3 with an unmodified Makefile (hence I suspect my custom compiler flags could trigger these issues on my system). Regardless of any modifications, I could not get a successfully compiled GCC-LTO-Kernel to boot yet. Without your patchset, a GCC-11.1 built Xanmod-Kernel compiles without problems with my custom Makefile and also boots successfully.

Edit: I've just noticed a segfault in dmesg which came from that ICE:

[Mai 1 19:17] cc1[112271]:segfault at fffffffffe0ed870 ip 00000000014a6ec8 sp 00007fff3a54f450 error 5 in cc1[59e000+f0d000]
[ +0,000013] Code: e4 ff ff 48 8b 44 24 08 49 89 84 24 a0 00 00 00 e9 28 fe ff ff 0f 1f 44 00 00 49 8b 94 24 98 00 00 00 49 8b 84 24 90 00 00 00 <80> 3a 48 0f 85 6f ff ff ff 48 b9 c7 c0 0f 00 00 00 0f 05 48 39 4a
dmesg.txt

[andikleen]

Does it boot for you without the xanmod kernel as base? And when it doesn't boot, what's the output?

I tested your config with gcc11 with my tree and it gets to mounting root at least.

For the ICEs, my bet would be on the on the graphite / loop-nest options. graphite is relatively buggy in gcc I believe, since it isn't really maintained. It would surprise me if the kernel had loops that benefited from it anyways. Perhaps try removing those.

I don't think those will have an impact: "-fno-math-errno -fno-trapping-math -mtls-dialect=gnu2", since the kernel neither uses floating point, nor compiler TLS.

Using -fno-semantic-interposition is an interesting idea. Perhaps should experiment with that,

[ms178]

Everything about my PC is somewhat experimental, even down to the compiler flags of the Kernel and the compiler used (got it from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/x86-64-v3). As I am just an enthusiast user, my experiences with these flags and where they benefit came down to some research what the Gentoo community does use somewhere and reading the GCC docs. So thanks for your insights to what might sense to use with the Kernel.

As mentioned in the top post, I did try the vanilla sources with your patchset, too - but I couldn't get that to boot on my machine yet either. I've also tried to build the Kernel without any modules but that leads to a non-booting Kernel which then shuts down the system after some seconds. I'll try out some further experiments today with the vanilla sources and will let you know about any progress.

[ms178]

Unfortunately, such a Kernel is still not booting and the issues as described in the first post in this thread are also still there:

Experiment 1 - Vanilla sources, custom config, custom flags in the Makefile - not booting

config.txt
Makefile.txt

Experiment 2 - Vanilla sources, custom config, everything else at stock values - not booting

config.txt
Makefile.txt

[andikleen]

The config boots for me in a KVM VM upto the root mount.
I assume there is some output before it fails.
Can you post it? Photo or video are ok.

[ms178]

There is nothing much to see, it is just a black page with the Kernel string on the first line and something like "Loading Kernel from Ramdisk" on the second line, followed by a static underscore. The only thing I can notice else is that the lighting of the keyboard goes off at that moment as well. I usually reset the system manually after a minute or so because nothing else seems to happen from that moment on.

By the way, I have multiple hard disks in the system, the Linux system is on a RAID-0 array consisting of two 2 TB SSHDs, there are also a single 500 GB SSD and a 1 TB NVMe PCIe 3.0 device in the system (but that shouldn't be a problem as all other Kernels boot fine from the RAID array). I've also tried GZIP, ZSTD and LZ4 compression of the Kernel as there was a post on LKML that warned of a similar boot issue with GCC 11.1 and GZIP (see: http://lkml.iu.edu/hypermail/linux/kernel/2104.3/06346.html) but my config file doesn't have the mentioned CONFIG_INITRAMFS_COMPRESSION_GZIP entry nor does O2 help me and I usually set it to XZ, ZSTD or LZ4 anyway. Also I can get a GCC 11.1-compiled Xanmod Kernel (which uses O3 by default with either XZ, ZSTD or LZ4 set) to boot without your patch set.

Do you also use the same GCC compiler version gcc version 11.1.0 (Ubuntu 11.1.0-1ubuntu1~21.04+v3 from https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/x86-64-v3) as I do? Maybe the combination of the GCC toolchain I use and your patchset trigger the issue.

[andikleen]

Do you see any output with earlyprintk=vga ? You may need to remove any vga=... arguments too to see the output.

No I don't use the ubuntu packages, but my own built gccs.

[ms178]

I am afraid, neither earlyprintk=vga nor earlyprintk do show me something at all [whereas I can see the boot details with my LLVM-LTO Kernel].

Here is a screenshot: