[GR-44565] Use the system libyaml for psych as it is faster for warmup

PullRequest: truffleruby/3673

Author: eregon

CHANGELOG.md

@@ -99,6 +99,7 @@ Performance:
 * Marking of native structures wrapped in objects is now done on C call exit to reduce memory overhead (@aardvark179).
 * Splitting (copying) of call targets has been optimized by implementing `cloneUninitialized()` (@andrykonchin, @eregon).
 * `Process.pid` is now cached per process like `$$` (#2882, @horakivo)
+* Use the system `libyaml` for `psych` to improve warmup when parsing YAML (#2089, @eregon).
 
 Changes:
 

README.md

@@ -103,6 +103,7 @@ environment, for example, by unmounting system filesystems such as `/dev/shm`.
 
 * [make and gcc](doc/user/installing-llvm.md) for building C and C++ extensions
 * [libssl](doc/user/installing-libssl.md) for the `openssl` C extension
+* [libyaml](doc/user/installing-libyaml.md) for the `psych` C extension
 * [zlib](doc/user/installing-zlib.md) for the `zlib` C extension
 
 Without these dependencies, many libraries including RubyGems will not work.

ci.jsonnet

@@ -228,12 +228,14 @@ local part_definitions = {
       packages+: {
         binutils: ">=2.30",
         ruby: "==" + mri_version,
+        libyaml: "==0.2.5",
       },
     },
 
     local linux_aarch64_deps = common.deps.sulong + {
       packages+: {
         ruby: "==3.0.2",
+        libyaml: "==0.2.5",
       },
     },
 

doc/user/installing-libssl.md

@@ -17,7 +17,7 @@ To compile TruffleRuby against a non-system `libssl`, set `OPENSSL_PREFIX` while
 export OPENSSL_PREFIX=/path/to/my/openssl-1.1.0
 ```
 
-### RedHat-based: Fedora, Oracle Linux, etc
+### Fedora-based: RHEL, Oracle Linux, etc
 
 ```bash
 sudo dnf install openssl-devel

doc/user/installing-libyaml.md

@@ -0,0 +1,42 @@
+---
+layout: docs-experimental
+toc_group: ruby
+link_title: Installing LibYAML
+permalink: /reference-manual/ruby/InstallingLibYAML/
+---
+# Installing LibYAML
+
+TruffleRuby requires to have `libyaml` installed, much like CRuby 3.2+ and Psych 5+.
+
+If you experience `psych`-related errors saying it cannot find `libyaml`, it might help to recompile the `psych` gem by running `lib/truffle/post_install_hook.sh`.
+This is done automatically by Ruby managers, and mentioned in the post-install message when installing TruffleRuby via `gu install` in GraalVM.
+
+### Fedora-based: RHEL, Oracle Linux, etc
+
+```bash
+sudo dnf install libyaml-devel
+```
+
+### Debian-based: Ubuntu, etc
+
+```bash
+sudo apt-get install libyaml-dev
+```
+
+### macOS
+
+#### Homebrew
+
+We recommend installing libssl via [Homebrew](https://brew.sh).
+
+```bash
+brew install libyaml
+```
+
+#### MacPorts
+
+MacPorts should also work but is not actively tested.
+
+```bash
+sudo port install libyaml
+```

doc/user/installing-llvm.md

@@ -12,19 +12,19 @@ If you are using an older version, see [the documentation for that version](http
 
 The `make` utility as well as the dependencies of the toolchain still need to be available to build C and C++ extensions.
 
-## RedHat-based: Fedora, Oracle Linux, etc.
+### Fedora-based: RHEL, Oracle Linux, etc
 
 ```bash
 sudo dnf install make gcc
 ```
 
-## Debian-based: Ubuntu, etc.
+### Debian-based: Ubuntu, etc
 
 ```bash
 sudo apt-get install make gcc
 ```
 
-## Mandriva-based and other Linux distributions
+### Mandriva-based and other Linux distributions
 
 Note: Such distributions are not tested and not [supported](../../README.md#system-compatibility).
 
@@ -40,7 +40,7 @@ cd /usr/lib/gcc
 sudo ln -s x86_64-mandriva-linux-gnu x86_64-linux-gnu
 ```
 
-## macOS
+### macOS
 
 On macOS, make sure you have installed the command line developer tools from Xcode:
 

doc/user/installing-zlib.md

@@ -8,7 +8,7 @@ permalink: /reference-manual/ruby/Installingzlib/
 
 TruffleRuby provides the `zlib` module but not the native `zlib` system library that the module uses.
 
-### RedHat-based: Fedora, Oracle Linux, etc
+### Fedora-based: RHEL, Oracle Linux, etc
 
 ```bash
 sudo dnf install zlib-devel

doc/user/utf8-locale.md

@@ -18,7 +18,7 @@ locale
 shows no `="C"` and no warning.
 Instead, all values should be `"en_US.UTF-8"` or other regions but still `.UTF-8`.
 
-### RedHat-based: Fedora, Oracle Linux, etc
+### Fedora-based: RHEL, Oracle Linux, etc
 
 ```bash
 export LANG=en_US.UTF-8

lib/truffle/post_install_hook.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 # Hook run after extraction in the installation directory by a Ruby installer.
 # Useful to perform tasks that depend on the user machine and
-# cannot be generally done in advance before building the release tarball.
+# cannot be generally done in advance on another machine.
 
 set -e
 
@@ -15,24 +15,38 @@ export TRUFFLERUBYOPT="--disable-gems $TRUFFLERUBYOPT"
 
 cd "$root"
 
-function recompile_openssl() {
-  cd src/main/c/openssl
+function recompile() {
+  ext="$1"
+  cd "src/main/c/$ext"
   truffleruby -w extconf.rb
   if [ -z "$CORES" ]; then
     CORES=$(getconf _NPROCESSORS_ONLN || echo 1)
   fi
   make "--jobs=$CORES"
-  cp "openssl.$(truffleruby -rrbconfig -e "print RbConfig::CONFIG['DLEXT']")" "$root/lib/mri"
+  cp "$ext.$(truffleruby -rrbconfig -e "print RbConfig::CONFIG['DLEXT']")" "$root/lib/mri"
+  cd "$root"
+  echo
 }
 
 if [ "$TRUFFLERUBY_RECOMPILE_OPENSSL" == "false" ]; then
-  echo "Skipping recompilation of the OpenSSL extension (TRUFFLERUBY_RECOMPILE_OPENSSL=false)"
+  echo "Skipping recompilation of the OpenSSL C extension (TRUFFLERUBY_RECOMPILE_OPENSSL=false)"
 elif [ "$TRUFFLERUBY_RECOMPILE_OPENSSL" == "true" ]; then
   echo "Recompiling the OpenSSL C extension (TRUFFLERUBY_RECOMPILE_OPENSSL=true)"
-  recompile_openssl
+  recompile openssl
 else
   echo "Recompiling the OpenSSL C extension (against the installed libssl)"
-  recompile_openssl
+  recompile openssl
+fi
+
+# Recompiling psych is needed on macOS if a non-default Homebrew prefix or not using Homebrew (e.g. MacPorts)
+if [ "$TRUFFLERUBY_RECOMPILE_PSYCH" == "false" ]; then
+  echo "Skipping recompilation of the Psych C extension (TRUFFLERUBY_RECOMPILE_PSYCH=false)"
+elif [ "$TRUFFLERUBY_RECOMPILE_PSYCH" == "true" ]; then
+  echo "Recompiling the Psych C extension (TRUFFLERUBY_RECOMPILE_PSYCH=true)"
+  recompile psych
+else
+  echo "Recompiling the Psych C extension (against the installed libyaml)"
+  recompile psych
 fi
 
 echo "TruffleRuby was successfully installed in $root"

lib/truffle/truffle/libyaml-prefix.rb

@@ -0,0 +1,41 @@
+# Copyright (c) 2023 Oracle and/or its affiliates. All rights reserved. This
+# code is released under a tri EPL/GPL/LGPL license. You can use it,
+# redistribute it and/or modify it under the terms of the:
+#
+# Eclipse Public License version 2.0, or
+# GNU General Public License version 2, or
+# GNU Lesser General Public License version 2.1.
+
+# Set LIBYAML_PREFIX in ENV to find the libyaml headers
+
+search_homebrew = -> homebrew {
+  if prefix = "#{homebrew}/opt/libyaml" and Dir.exist?(prefix)
+    prefix
+  end
+}
+
+if Truffle::Platform.darwin? && !ENV['LIBYAML_PREFIX']
+  default_homebrew_prefix = Truffle::System.host_cpu == 'aarch64' ? '/opt/homebrew' : '/usr/local'
+  if prefix = search_homebrew.call(default_homebrew_prefix)
+    # found
+  else
+    homebrew = `brew --prefix 2>/dev/null`.strip
+    homebrew = nil unless $?.success? and !homebrew.empty? and Dir.exist?(homebrew)
+
+    if homebrew and prefix = search_homebrew.call(homebrew)
+      # found
+    elsif Dir.exist?('/opt/local/include/libyaml') # MacPorts
+      prefix = '/opt/local'
+    end
+  end
+
+  if prefix
+    ENV['LIBYAML_PREFIX'] = prefix
+  else
+    abort 'Could not find libyaml headers, install via Homebrew or MacPorts or set LIBYAML_PREFIX'
+  end
+end
+
+if libyaml_prefix = ENV['LIBYAML_PREFIX']
+  Truffle::Debug.log_config("Found libyaml in #{libyaml_prefix}")
+end