Help Building .NET 8 on RHEL7

[n00j]

Due to some legacy reasons my company is stuck on RHEL 7 for a little while. We have previously been using the prebuilt binaries which were working fine. Going forward it looks like .NET 8 will be dropping RHEL 7 support so I was looking to build from source.

I was hoping to get some guidance on how I would go about building .NET 8 using these instructions https://github.com/dotnet/dotnet#building

It looks to me like the prebuilt artifacts the build currently relies on also requires RHEL 8, and the build fails when I attempt to build on RHEL 7. If anyone has some pointers on how to build .NET 8 on RHEL 7, that would be very helpful. Is it possible to use .NET 7 or .NET 6 to build .NET 8 on RHEL 7?

[omajid]

I don't expect this to work easily, for a few reasons.

• Building .NET 8 will require a recent-ish pre-release of .NET 8. But as you are aware, we don't quite have a working .NET 8 on RHEL 7 to build with. You might be able to work around this by building on another distro but providing a RHEL-7-compatible glibc with a rootfs. You can then take that .NET build to RHEL 7 and then use it to build future versions of .NET on RHEL 7. The rootfs approach is supposed to work - Microsoft builds .NET this way - but it hasn't been tested in a source-build context ever, I think.
• .NET 8 requires a bunch of packages (eg, cmake) with higher versions than what is available on RHEL 7. You might be able to find newer versions built by someone else. (Then again, this moves you off of the supported path on RHEL 7, if you care about that sort of thing)

Is it possible to use .NET 7 or .NET 6 to build .NET 8 on RHEL 7?

I wish it was. It isn't :(

Due to some legacy reasons my company is stuck on RHEL 7 for a little while.

Have you considered using a container that carries the RHEL 8 userspace but can run on RHEL 7?

[n00j]

Appreciate the response!

Building .NET 8 will require a recent-ish pre-release of .NET 8. But as you are aware, we don't quite have a working .NET 8 on RHEL 7 to build with. You might be able to work around this by building on another distro but providing a RHEL-7-compatible glibc with a rootfs. You can then take that .NET build to RHEL 7 and then use it to build future versions of .NET on RHEL 7. The rootfs approach is supposed to work - Microsoft builds .NET this way - but it hasn't been tested in a source-build context ever, I think.

By rootfs approach, I assume you mean creating a chroot jail with the necessary dependencies? Will give this a shot, only issue I would see is I would need access to both version of glib for rhel7 (for the build of .net 8 for RHEL7), and rhel8 (for running the bootstrap sdk).

.NET 8 requires a bunch of packages (eg, dotnet/runtime#74027) with higher versions than what is available on RHEL 7. You might be able to find newer versions built by someone else. (Then again, this moves you off of the supported path on RHEL 7, if you care about that sort of thing)

Funnily enough, we do this already internally so most of the dependencies required already work internally, it is only the bootstrapping sdk that doesn't run

Have you considered using a container that carries the RHEL 8 userspace but can run on RHEL 7?

Yes, that would be the best solution. Unfortunately we have some deployment targets that don't support container deployments, so was hoping to find a way to support those as well.

[omajid]

By rootfs, I meant the approach described here: https://github.com/dotnet/runtime/blob/main/docs/workflow/building/coreclr/cross-building.md#linux-cross-building.

You would have do this on RHEL 8 and create a RHEL 7 rootfs. I am not sure how to manually do that.

You would also need to build the rest of the SDK and associated nuget packages and then carry them over across builds. There's an example at https://github.com/ppc64le/build-scripts/blob/master/d/dotnet7/dotnet-build-ppc64le that uses a rootfs to build for ppc64le on an x86_64 machine; your scenario would be similar, just building for x86_64 (old glibc) on x86_64 (with new glibc).

[n00j]

Just an update on this in case anyone else is trying this. I was able to create a rootfs for RHEL7 on WSL and build the .net 8 runtime successfully. Next step to build the sdk.

The easiest way to build the rootfs I found, was to create a docker image with RHEL7 base install the necessary dependencies (build llvm from source https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-16.0.6.tar.gz), install an updated gcc via the yum package devtoolset-12.
Then flatten the image into a single layer via https://stackoverflow.com/questions/22713551/how-to-flatten-a-docker-image
Then extract out the layer into the file system using docker save, and extract out the tar file.

[omajid]

Nice! Would you be able to share your scripts to do this?

[n00j]

Yes, let me clean this up and I'll see if I can post that.

[n00j]

Here are the scripts I used to build the sdk for rhel7:
https://github.com/n00j/dotnet8centos7

I switched these to centos, but i'm not sure this was the best idea. Packages available in rhel7/8 yum aren't available in centos, but the general idea is there. After running the build.sh script I manually combined the runtime + sdk tar.gz files and was able to run them on rhel7 to build + run a .net 8 project.

I'm not sure on the next steps. Reading this: building-for-new-architecture-using-a-rid-unknown-to-net, I'm supposed to now follow: [Use the cross-compiled SDK and augmented previously-source-built-archive to build a stage 1 SDK. See the Stage 1 instructions in Building on a supported platform (Using RID known to .NET).

Using the --with-sdk flag, but this doesn't work as i'm not sure what augmented previously-source-built-archive exactly means.

[omajid]

134 generally means a SIGABRT. Can get a coredump and/or run a backtrace?

[n00j]

I got the coredump, and I opened it with gdb:

Missing separate debuginfo for /sdk/host/fxr/8.0.0-rc.2.23479.6/libhostfxr.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f0/bc45fe9254789994d8c17ee19bc7097d11c6af.debug
Missing separate debuginfo for /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libhostpolicy.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/1a/9261c4f0c3bf5a53f26e5c67616fd7572af287.debug
Missing separate debuginfo for /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/4c/08d0921f5508ffdd2326f5e460f0e7892e564b.debug
Missing separate debuginfo for /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclrtraceptprovider.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f7/3e46df5d2ed3bf1a740c77c392eb00d0bd490e.debug
Missing separate debuginfo for /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libclrjit.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/79/ee05e7398e1d9792d2473c2d18434bf5d42ffb.debug
Missing separate debuginfo for /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libSystem.Native.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/70/0e2dc96059f267d8900e928c6e68240b42d5f4.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/sdk/dotnet /dotnet/src/fsharp/artifacts/source-build/self/src/artifacts/Bootst'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f158c586387 in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f158d770780 (LWP 13173))]
Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libicu-50.2-4.el7_7.x86_64 libstdc++-4.8.5-44.el7.x86_64 lttng-ust-2.4.1-4.el7.x8--Type <RET> for more, q to quit, c to continue without paging-- c

[n00j]

not sure I have any debug symbols...

(gdb) backtrace
#0  0x00007f158c586387 in raise () from /lib64/libc.so.6
#1  0x00007f158c587a78 in abort () from /lib64/libc.so.6
#2  0x00007f158c478a47 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#3  0x00007f158c478969 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#4  0x00007f158c15d17f in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#5  0x00007f158c01e13e in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#6  0x00007f158c01e376 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#7  0x00007f158c01e4c4 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#8  0x00007f158c00f4cc in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#9  0x00007f158c00c3de in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#10 0x00007f158c02c063 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#11 0x00007f158c01a1ec in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#12 0x00007f158c03d7c2 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#13 0x00007f158c0e2757 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#14 0x00007f158c0e0f61 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#15 0x00007f158c094d1c in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#16 0x00007f158460a6be in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libclrjit.so
#17 0x00007f1584563c21 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libclrjit.so
#18 0x00007f158456279f in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libclrjit.so
#19 0x00007f158456486b in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libclrjit.so
#20 0x00007f1584569666 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libclrjit.so
#21 0x00007f158c09831b in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#22 0x00007f158c098523 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#23 0x00007f158c0990ce in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#24 0x00007f158c0d3621 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#25 0x00007f158c0d2f02 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#26 0x00007f158c0d2690 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#27 0x00007f158c0d209e in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#28 0x00007f158c045c57 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#29 0x00007f158c0d64a5 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#30 0x00007f158c0d6079 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#31 0x00007f158c2f2ea0 in ?? () from /sdk/shared/Microsoft.NETCore.App/8.0.0-rc.2.23479.6/libcoreclr.so
#32 0x00007f15108c5612 in ?? ()
#33 0xfd25ff00003ffb15 in ?? ()
#34 0x000000012143fae8 in ?? ()
#35 0x000000000000000c in ?? ()
#36 0x00000e878d770ab0 in ?? ()
#37 0xffffffff0000000c in ?? ()
#38 0x00007ed52141fbc0 in ?? ()
#39 0x00007ed521441a88 in ?? ()
#40 0x00007ed521441a88 in ?? ()
#41 0x00007ed52141ecc8 in ?? ()
#42 0x00007ed52143fdc0 in ?? ()
#43 0x0000000121441a70 in ?? ()
#44 0x00007ed52143fae8 in ?? ()
#45 0x00007ffe76dc9d90 in ?? ()
#46 0x00007f15108c54c4 in ?? ()
#47 0x0000000000000000 in ?? ()

[n00j]

Is there a way to skip building fsharp? We actually don't need it atm, and if I could just skip that part for now, that would be great.

[omajid]

not sure I have any debug symbols...

Did you build this RC 2 SDK yourself? If so, they should have included debug symbols. If you used a prebuilt SDK from Microsoft, you can use https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-symbol and lldb to get better stack traces.

Is there a way to skip building fsharp?

Not really. We can remove fsharp from the build configuration, but the .NET SDK has a dependency on it. So it might pull down a version built by Microsoft from nuget.org, or might just outright fail the build.

[n00j]

not sure I have any debug symbols...

Did you build this RC 2 SDK yourself? If so, they should have included debug symbols. If you used a prebuilt SDK from Microsoft, you can use https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-symbol and lldb to get better stack traces.

Is there a way to skip building fsharp?

Not really. We can remove fsharp from the build configuration, but the .NET SDK has a dependency on it. So it might pull down a version built by Microsoft from nuget.org, or might just outright fail the build.

I built the rc2 bootstrap version myself, but I might not have saved the symbols 😭

I just put exit 0 in the fsharp build.sh script and it seems to have bypassed it, and it seems to have succeeded the build. I also had to delete the prebuilt artifacts the source-build downloaded that were referencing centos.8.* e.g. rm /prebuilt/*.centos.* # remove and centos 8 + packages that were downloaded

I just tested a simple hello world on a rhel7 machine and it looks like that worked. Will do some more testing 🤞

> dotnet new console
> dotnet run
Hello, World!

Thanks for all the help!