No longer ship the GraalVM LLVM Toolchain as it is no longer needed

eregon · eregon · commit bd659a9a6bb8 · 2023-11-29T16:10:31.000+01:00
diff --git a/ci.jsonnet b/ci.jsonnet
@@ -368,6 +368,7 @@ local part_definitions = {
     },
 
     test_cexts_sulong: {
+      mx_env:: "toolchain",
       environment+: {
         TRUFFLERUBYOPT: "--experimental-options --cexts-sulong",
       },
diff --git a/doc/contributor/docker.md b/doc/contributor/docker.md
@@ -50,8 +50,6 @@ jt docker test --filter debian --standalone $PATH_TO_STANDALONE_TAR_GZ --test re
 For example, to run a full set of tests on a set of new release candidate tarballs:
 
 ```bash
-jt docker test --graalvm graalvm-ce.tar.gz llvm-toolchain-installable.jar:ruby-installable-ce.jar --test release_branch
-jt docker test --graalvm graalvm-ee.tar.gz llvm-toolchain-installable.jar:ruby-installable-ee.jar --test release_branch
 jt docker test --standalone truffleruby-linux-amd64.tar.gz --test release_branch
 ```
 
@@ -66,10 +64,7 @@ Pick one of the distributions in [docker-configs.yaml](../../tool/docker-configs
 
 ## Methods of installing
 
-Pick one of:
-
-* From a GraalVM binary tarball and Ruby component you have locally, `--graalvm graalvm.tar.gz llvm-toolchain-installable.jar:ruby-installable.jar`
-* From a TruffleRuby standalone distribution you have locally, `--standalone truffleruby-1.0.0-linux-amd64.tar.gz`
+From a TruffleRuby standalone distribution you have locally, `--standalone truffleruby-1.0.0-linux-amd64.tar.gz`
 
 ## What to do
 
diff --git a/doc/contributor/workflow.md b/doc/contributor/workflow.md
@@ -127,20 +127,6 @@ This warning is important.
 - If you want to set the `suite.py` import to that checked out in `graal` (unlikely), you should run 
   jt mx scheckimports` beforehand.
 
-### Building C Extensions more quickly
-
-Speeding up compilation of C extensions can be achieved by using *native* toolchain launchers,
-instead of the Bash toolchain launchers used with the `jvm*` build configurations.
-
-For bundled C extensions (that is, extensions under `src/main/c`),
-you can `export JT_CACHE_TOOLCHAIN=true` and that will then use native toolchain launchers for the
-[bootstrap toolchain](https://github.com/oracle/graal/blob/master/sulong/docs/contributor/TOOLCHAIN.md#using-a-prebuilt-graalvm-as-a-bootstrapping-toolchain).
-This has no effect after `jt build`.
-
-For C extensions installed after `jt build` by e.g. `gem install` or `bundle install`,
-one can use the `jvm-ce-ntl` build configuration which includes *native* toolchain launchers,
-or use one of the `native*` build configuration (which also builds a native truffleruby launcher).
-
 ## Editors and IDEs
 
 * [Using the IntelliJ IDE for Development](using-intellij.md) (recommended)
diff --git a/doc/user/installing-llvm.md b/doc/user/installing-llvm.md
@@ -6,11 +6,8 @@ permalink: /reference-manual/ruby/InstallingLLVM/
 ---
 # Installing Make and GCC
 
-Since TruffleRuby 19.3.0, TruffleRuby ships with its own LLVM toolchain.
-Therefore, it is no longer necessary to install LLVM.
-If you are using an older version, see [the documentation for that version](https://github.com/oracle/truffleruby/blob/vm-19.2.0/doc/user/installing-llvm.md).
-
-The `make` utility as well as the dependencies of the toolchain still need to be available to build C and C++ extensions.
+Since TruffleRuby 24.0.0, TruffleRuby no longer needs a LLVM toolchain and instead uses the system toolchain.
+The packages below are required to build C and C++ extensions.
 
 ### Fedora-based: RHEL, Oracle Linux, etc
 
@@ -24,28 +21,10 @@ sudo dnf install make gcc
 sudo apt-get install make gcc
 ```
 
-### Mandriva-based and other Linux distributions
-
-Note: Such distributions are not tested and not [supported](../../README.md#system-compatibility).
-
-First, install the `make` and `gcc` dependencies.
-
-Mandriva uses a not-yet-upstreamed patch to let `clang` find the GCC installation (see [this comment](https://github.com/oracle/truffleruby/issues/2009#issuecomment-630019082)).
-Therefore the internal LLVM toolchain cannot find the necessary `libgcc_s` by default.
-The proper fix is for those distributions to upstream their changes to LLVM.
-
-A workaround is to create a symlink explicitly so that the LLVM toolchain can find `libgcc_s`:
-```bash
-cd /usr/lib/gcc
-sudo ln -s x86_64-mandriva-linux-gnu x86_64-linux-gnu
-```
-
 ### macOS
 
 On macOS, make sure you have installed the command line developer tools from Xcode:
 
 ```bash
 xcode-select --install
 ```
-
-You might need to add `export SDKROOT=$(xcrun --show-sdk-path)` in your shell profile.
diff --git a/doc/user/polyglot.md b/doc/user/polyglot.md
@@ -28,7 +28,7 @@ The JVM configuration automatically has access to other languages.
 To use other GraalVM languages, you need the [JVM Standalone](../../README.md#getting-started).
 The Native Standalone does not support installing extra languages.
 
-Note that `ruby`, `llvm`, the LLVM toolchain and host Java interop are available without installing anything extra.
+Note that `ruby`, `llvm` and host Java interop are available without installing anything extra.
 
 Then you can install other languages with `truffleruby-polyglot-get $LANGUAGE`, for instance:
 ```bash
diff --git a/lib/truffle/rbconfig.rb b/lib/truffle/rbconfig.rb
@@ -119,8 +119,6 @@ module RbConfig
   cext_dir = "#{prefix}/lib/cext"
   soext = Truffle::Platform::SOEXT
 
-  toolchain_path = Truffle::Boot.toolchain_paths(:PATH)
-
   # Make C extensions use the same libssl as the one used for the openssl C extension
   if Truffle::Platform.darwin?
     require 'truffle/openssl-prefix'
@@ -225,7 +223,6 @@ module RbConfig
     'sysconfdir'        => "#{prefix}/etc", # doesn't exist, as in MRI
     'target_cpu'        => host_cpu,
     'target_os'         => host_os,
-    'toolchain_path'    => toolchain_path,
     'UNICODE_VERSION'   => Primitive.encoding_unicode_version,
     'UNICODE_EMOJI_VERSION' => Primitive.encoding_unicode_emoji_version,
     'warnflags'         => warnflags,
diff --git a/mx.truffleruby/env_files.md b/mx.truffleruby/env_files.md
@@ -1,16 +1,14 @@
 Here is how the various env files relate to each other:
 * `jvm`
   * `jvm-ce`: + GraalVM Community Compiler
-    * `jvm-ce-ntl`: + native toolchain launchers
-      * `jvm-ce-libgraal`: + libgraal
-      * `native`: + librubyvm + `Truffle Macro`
-        * `native-host-inlining`: + `TruffleHostInliningPrintExplored`, - native toolchain launchers
-        * `native-profiling`: + `-H:-DeleteLocalSymbols`
+    * `jvm-ce-libgraal`: + libgraal
+    * `native`: + librubyvm + `Truffle Macro`
+      * `native-host-inlining`: + `TruffleHostInliningPrintExplored`
+      * `native-profiling`: + `-H:-DeleteLocalSymbols`
   * `jvm-ee`: + Oracle GraalVM Compiler + `Truffle enterprise` + license + `LLVM Runtime Native Enterprise`
-    * `jvm-ee-ntl`: + native toolchain launchers
-      * `jvm-ee-libgraal`: + libgraal
-      * `native-ee`: + librubyvm + `Truffle Macro Enterprise` + Native Image G1
-        * `native-ee-host-inlining`: + `TruffleHostInliningPrintExplored`, - native toolchain launchers
-        * `native-ee-aux`: + `AuxiliaryEngineCache`, - Native Image G1 (currently incompatible)
+    * `jvm-ee-libgraal`: + libgraal
+    * `native-ee`: + librubyvm + `Truffle Macro Enterprise` + Native Image G1
+      * `native-ee-host-inlining`: + `TruffleHostInliningPrintExplored`
+      * `native-ee-aux`: + `AuxiliaryEngineCache`, - Native Image G1 (currently incompatible)
   * `jvm-js`: + Graal.js
   * `jvm-py`: + GraalPython
diff --git a/mx.truffleruby/jvm-ce-libgraal b/mx.truffleruby/jvm-ce-libgraal
@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/compiler,/substratevm
 COMPONENTS=TruffleRuby,suite:tools,GraalVM compiler,SubstrateVM,LibGraal
-NATIVE_IMAGES=suite:sulong,lib:jvmcicompiler
+NATIVE_IMAGES=lib:jvmcicompiler
 # To also create the standalone
 INSTALLABLES=TruffleRuby
 BUILD_TARGETS=GRAALVM,GRAALVM_STANDALONES
diff --git a/mx.truffleruby/jvm-ce-ntl b/mx.truffleruby/jvm-ce-ntl
diff --git a/mx.truffleruby/jvm-ee-libgraal b/mx.truffleruby/jvm-ee-libgraal
@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/truffleruby-enterprise,/graal-enterprise,/vm-enterprise,/sulong-managed,/substratevm-enterprise
 COMPONENTS=TruffleRuby,suite:tools,GraalVM enterprise compiler,Truffle enterprise,GraalVM enterprise license files,LLVM Runtime Native Enterprise,SubstrateVM Enterprise,LibGraal Enterprise
-NATIVE_IMAGES=suite:sulong,lib:jvmcicompiler
+NATIVE_IMAGES=lib:jvmcicompiler
 # To also create the standalone
 INSTALLABLES=TruffleRuby
 BUILD_TARGETS=GRAALVM,GRAALVM_STANDALONES
diff --git a/mx.truffleruby/jvm-ee-ntl b/mx.truffleruby/jvm-ee-ntl
diff --git a/mx.truffleruby/mx_truffleruby.py b/mx.truffleruby/mx_truffleruby.py
@@ -247,10 +247,15 @@ def ruby_maven_deploy_public(args):
     stability="experimental",
 ))
 
+# Only keep what we need from the Sulong home.
+# Exclude the toolchain launchers, we don't need them, and they would not work in the native standalone.
+# Excluding "native/bin" or "native/bin/*" does not work so we have to list them.
+toolchain_launchers = ['binutil', 'clang', 'clang++', 'clang-cl', 'flang', 'ld']
+sulong_home_excludes = [f"native/bin/graalvm-native-{launcher}" for launcher in toolchain_launchers] + \
+    ['native/cmake', 'native/include', 'native/lib/*++*']
 standalone_dependencies_common = {
     'LLVM Runtime Core': ('lib/sulong', []),
-    'LLVM Runtime Native': ('lib/sulong', []),
-    'LLVM.org toolchain': ('lib/llvm-toolchain', ['bin/flang-*', 'bin/f18*', 'lib/libFortran*', 'include/flang']),
+    'LLVM Runtime Native': ('lib/sulong', sulong_home_excludes),
 }
 
 mx_sdk_vm.register_graalvm_component(mx_sdk_vm.GraalVmLanguage(
@@ -262,7 +267,7 @@ def ruby_maven_deploy_public(args):
     standalone_dir_name_enterprise='truffleruby-<version>-<graalvm_os>-<arch>',
     license_files=[],
     third_party_license_files=[],
-    dependencies=['rbyl', 'Truffle', 'Truffle NFI', 'LLVM Runtime Native', 'LLVM.org toolchain', 'TRegex'],  # Use short name for license to select by priority
+    dependencies=['rbyl', 'Truffle', 'Truffle NFI', 'LLVM Runtime Native', 'TRegex'],  # Use short name for license to select by priority
     standalone_dependencies={**standalone_dependencies_common, **{
         'TruffleRuby license files': ('', []),
     }},
diff --git a/mx.truffleruby/native b/mx.truffleruby/native
@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/compiler,/substratevm
 COMPONENTS=TruffleRuby,suite:tools,GraalVM compiler,SubstrateVM,Truffle Macro
-NATIVE_IMAGES=suite:sulong,lib:rubyvm
+NATIVE_IMAGES=lib:rubyvm
 EXTRA_IMAGE_BUILDER_ARGUMENTS=rubyvm:-H:+UnlockExperimentalVMOptions rubyvm:-H:BuildOutputJSONFile=native-image-build-rubyvm.json rubyvm:-H:-UnlockExperimentalVMOptions
 GENERATE_DEBUGINFO=false
 # To also create the standalone
diff --git a/mx.truffleruby/native-ee b/mx.truffleruby/native-ee
@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/truffleruby-enterprise,/graal-enterprise,/vm-enterprise,/sulong-managed,/substratevm-enterprise,substratevm-enterprise-gcs
 COMPONENTS=TruffleRuby,suite:tools,GraalVM enterprise compiler,Truffle enterprise,GraalVM enterprise license files,LLVM Runtime Native Enterprise,SubstrateVM Enterprise,Truffle Macro Enterprise,suite:substratevm-enterprise-gcs
-NATIVE_IMAGES=suite:sulong,lib:rubyvm
+NATIVE_IMAGES=lib:rubyvm
 EXTRA_IMAGE_BUILDER_ARGUMENTS=rubyvm:-H:+UnlockExperimentalVMOptions rubyvm:-H:BuildOutputJSONFile=native-image-build-rubyvm.json rubyvm:-H:-UnlockExperimentalVMOptions
 GENERATE_DEBUGINFO=false
 # To also create the standalone
diff --git a/mx.truffleruby/native-ee-aux b/mx.truffleruby/native-ee-aux
@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/truffleruby-enterprise,/graal-enterprise,/vm-enterprise,/sulong-managed,/substratevm-enterprise
 COMPONENTS=TruffleRuby,suite:tools,GraalVM enterprise compiler,Truffle enterprise,GraalVM enterprise license files,LLVM Runtime Native Enterprise,SubstrateVM Enterprise,Truffle Macro Enterprise
-NATIVE_IMAGES=suite:sulong,lib:rubyvm
+NATIVE_IMAGES=lib:rubyvm
 EXTRA_IMAGE_BUILDER_ARGUMENTS=rubyvm:-H:+UnlockExperimentalVMOptions rubyvm:-H:BuildOutputJSONFile=native-image-build-rubyvm.json rubyvm:-H:-UnlockExperimentalVMOptions rubyvm:-H:+AuxiliaryEngineCache rubyvm:-H:ReservedAuxiliaryImageBytes=1073741824
 GENERATE_DEBUGINFO=false
 NATIVE_IMAGE_AUXILIARY_ENGINE_CACHE=true
diff --git a/mx.truffleruby/native-profiling b/mx.truffleruby/native-profiling
@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/compiler,/substratevm
 COMPONENTS=TruffleRuby,suite:tools,GraalVM compiler,SubstrateVM,Truffle Macro
-NATIVE_IMAGES=suite:sulong,lib:rubyvm
+NATIVE_IMAGES=lib:rubyvm
 EXTRA_IMAGE_BUILDER_ARGUMENTS=rubyvm:-H:+UnlockExperimentalVMOptions rubyvm:-H:BuildOutputJSONFile=native-image-build-rubyvm.json rubyvm:-H:-DeleteLocalSymbols rubyvm:-H:-UnlockExperimentalVMOptions
 GENERATE_DEBUGINFO=false
 # To also create the standalone
diff --git a/mx.truffleruby/toolchain b/mx.truffleruby/toolchain
@@ -0,0 +1,4 @@
+GRAALVM_SKIP_ARCHIVE=true
+DYNAMIC_IMPORTS=/tools
+COMPONENTS=TruffleRuby,suite:tools,LLVM.org toolchain
+BUILD_TARGETS=GRAALVM
diff --git a/spec/truffle/standalone_spec.rb b/spec/truffle/standalone_spec.rb
@@ -10,12 +10,19 @@
 require_relative '../ruby/spec_helper'
 
 describe "The standalone" do
-  llvm_toolchain = "#{Truffle::Boot.ruby_home}/lib/llvm-toolchain"
-  guard -> { Dir.exist? llvm_toolchain } do
-    it "should not include Fortran-related files in the toolchain" do
-      Dir.glob("#{llvm_toolchain}/bin/flang*").should.empty?
-      Dir.glob("#{llvm_toolchain}/lib/libFortran*").should.empty?
-      Dir.glob("#{llvm_toolchain}/include/flang/*").should.empty?
+  sulong = "#{Truffle::Boot.ruby_home}/lib/sulong"
+
+  guard -> { Dir.exist? sulong } do # check if standalone, see standalone_dependencies
+    it "only includes the necessary Sulong libs" do
+      Dir.glob("#{sulong}/*").filter_map { File.basename(_1) if Dir.exist?(_1) }.should == %w[include native]
+      Dir.glob("#{sulong}/native/*").map { File.basename(_1) }.should == %w[lib]
+      Dir.glob("#{sulong}/native/lib/*") do
+        File.basename(_1).should_not.include?('++')
+      end
+    end
+
+    it "does not include the llvm toolchain" do
+      Dir.should_not.exist? "#{Truffle::Boot.ruby_home}/lib/llvm-toolchain"
     end
   end
 end
diff --git a/src/main/c/cext-trampoline/Makefile b/src/main/c/cext-trampoline/Makefile
@@ -7,10 +7,6 @@ else
 SOEXT := so
 endif
 
-# cc should be enough to compile this library but might produce different warnings based on the system compiler.
-# If that is the case and fails the build, it might be better to use instead:
-# CC := $(GRAALVM_TOOLCHAIN_CC)
-
 # Similar values as in RbConfig
 ifeq ($(OS),Darwin)
 CC := clang
diff --git a/src/options.yml b/src/options.yml
@@ -177,7 +177,7 @@ INTERNAL: # Options for debugging the TruffleRuby implementation
     CEXTS_TO_NATIVE_COUNT: [cexts-to-native-count, boolean, CEXTS_TO_NATIVE_STATS, Track the number of conversions of VALUEs to native]
     BACKTRACE_ON_TO_NATIVE: [backtraces-to-native, boolean, false, Show a backtrace when a ValueWrapper handle is created for a Ruby object]
     CEXTS_KEEP_HANDLES_ALIVE: [keep-handles-alive, boolean, false, Keep handles for value wrappers alive forever]
-    # jt clean cexts && TRUFFLERUBYOPT="--experimental-options --cexts-sulong" jt build && TRUFFLERUBYOPT="--experimental-options --cexts-sulong" jt test :capi
+    # jt clean cexts && TRUFFLERUBYOPT="--experimental-options --cexts-sulong" jt build --env toolchain && TRUFFLERUBYOPT="--experimental-options --cexts-sulong" jt -u toolchain test :cext
     CEXTS_SULONG: [cexts-sulong, boolean, false, "Run C extensions on Sulong instead of natively, requires rebuilding TruffleRuby from source with this option"]
 
     # Options to debug the implementation
diff --git a/test/truffle/cexts/backtraces/ext/backtraces/extconf.rb b/test/truffle/cexts/backtraces/ext/backtraces/extconf.rb
@@ -13,14 +13,11 @@ def command(*args)
 name = "#{dir}/libnativetest"
 so = RbConfig::CONFIG['SOEXT']
 
-original_path = ENV['PATH'].delete_prefix("#{RbConfig::CONFIG['toolchain_path']}:")
-system_cc = find_executable('cc', original_path)
-
 cflags = %w[-Wall -Werror -fPIC -std=c99]
 ldflags = %w[]
 
-command system_cc, '-o', "#{name}.o", '-c', *cflags, *ldflags, "#{name}.c"
-command system_cc, '-shared', *ldflags, '-o', "#{name}.#{so}", "#{name}.o"
+command 'cc', '-o', "#{name}.o", '-c', *cflags, *ldflags, "#{name}.c"
+command 'cc', '-shared', *ldflags, '-o', "#{name}.#{so}", "#{name}.o"
 
 $LIBS += " #{name}.#{so}"
 
diff --git a/tool/compare-rbconfig.rb b/tool/compare-rbconfig.rb
@@ -19,7 +19,6 @@
   f.puts RUBY_VERSION
 
   prefix = RbConfig::CONFIG['prefix']
-  toolchain_path = RbConfig::CONFIG['toolchain_path']
 
   {
     # 'RbConfig::CONFIG' => RbConfig::CONFIG,
@@ -31,7 +30,6 @@
       value = h[k]
 
       value = value.gsub(prefix, '$(prefix)')
-      value = value.gsub(toolchain_path, '$(toolchain_path)') if toolchain_path
 
       f.puts "#{k} = #{value}"
     }
diff --git a/tool/jt.rb b/tool/jt.rb
@@ -952,7 +952,6 @@ def clean(*options)
       sh 'rm', '-rf', 'spec/ruby/ext'
       Dir.glob("#{TRUFFLERUBY_DIR}/mxbuild/{*,.*}") do |path|
         next if File.basename(path).start_with?('truffleruby-')
-        next if File.basename(path).start_with?('toolchain')
         next if %w(. ..).include? File.basename(path)
         sh 'rm', '-rf', path
       end
@@ -2447,35 +2446,6 @@ def checkout_enterprise_revision(env = 'jvm-ee')
     mx('--env', env, 'checkout-downstream', 'compiler', 'graal-enterprise', primary_suite: TRUFFLERUBY_DIR)
   end
 
-  def bootstrap_toolchain
-    sulong_home = File.join(GRAAL_DIR, 'sulong')
-    # clone the graal repository if it is missing
-    mx 'sversions' unless File.directory? sulong_home
-    graal_version = get_truffle_version from: :repository
-    toolchain_dir = File.join(TRUFFLERUBY_DIR, 'mxbuild', 'toolchain')
-    destination = File.join(toolchain_dir, graal_version)
-    unless File.exist? destination
-      puts "Building toolchain for: #{graal_version}"
-      mx '--env', 'toolchain-only', 'build', primary_suite: sulong_home
-      toolchain_graalvm = mx('--env', 'toolchain-only', 'graalvm-home', primary_suite: sulong_home, capture: :out).lines.last.chomp
-      FileUtils.mkdir_p destination
-      FileUtils.cp_r toolchain_graalvm + '/.', destination
-    end
-
-    # mark as used
-    FileUtils.touch destination
-    # leave only 4 built toolchains which were used last
-    caches = Dir.
-        glob(File.join(toolchain_dir, '*')).
-        sort_by { |cached_toolchain_path| File.mtime cached_toolchain_path }
-    unless (oldest = caches[0...-4]).empty?
-      puts "Removing old cached toolchains: #{oldest.join ' '}"
-      FileUtils.rm_rf oldest
-    end
-
-    destination
-  end
-
   private def sforceimports?(mx_base_args)
     return true unless File.directory?(GRAAL_DIR)
 
@@ -2566,9 +2536,7 @@ def sforceimports
     mx_options, mx_build_options = args_split(options)
     mx_args = mx_base_args + mx_options
 
-    process_env = ENV['JT_CACHE_TOOLCHAIN'] ? { 'SULONG_BOOTSTRAP_GRAALVM' => bootstrap_toolchain } : {}
-
-    mx(process_env, *mx_args, 'build', *mx_build_options, primary_suite: TRUFFLERUBY_DIR)
+    mx(*mx_args, 'build', *mx_build_options, primary_suite: TRUFFLERUBY_DIR)
     build_dir = mx(*mx_args, 'graalvm-home', primary_suite: TRUFFLERUBY_DIR, capture: :out).lines.last.chomp
 
     dest = "#{TRUFFLERUBY_DIR}/mxbuild/#{name}"
