Handle --jvm flag in process with JNI instantiated JVM

[JaroslavTulach]

Pull Request Description

Fixes #12842 by designing a JNI OS environment, providing its implementations for supported platforms and using it to handle --jvm flag. All the Standard Library Tests (jvm) use this new launch-in-the-same-process technique.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• All code follows the
  Scala,
  Java,
• Unit tests have been written where possible.

[JaroslavTulach]

Now the problem is in executing Generic_JDBC_Tests - e.g. the switch to JVM mode isn't yet working correctly:

[org.enso.runner.JavaFinder] No appropriate JDK found in the distribution runtimes. Trying system-wide JDK.
Error occurred during initialization of boot layer
java.lang.UnsatisfiedLinkError: 'int sun.nio.fs.UnixNativeDispatcher.init()'

Unsatisfied link problem on Linux (update on May 7, 2025: got it working after removing java.desktop module from native image). We have linkage problem on Windows:

jvm.lib(jni.obj) : error LNK2005: JNI_GetDefaultJavaVMInitArgs already defined in enso.obj
jvm.lib(jni.obj) : error LNK2005: JNI_CreateJavaVM already defined in enso.obj
jvm.lib(jni.obj) : error LNK2005: JNI_GetCreatedJavaVMs already defined in enso.obj
net.lib(net_util_md.obj) : error LNK2005: DllMain already defined in jvm.lib(os_windows.obj)

Update on May 2, 2025: the Windows version works on GraalVM for JDK21, but not on any newer version!

On Mac the --jvm option already works OK.

[JaroslavTulach]

Linux

Trying to isolate the UnsatisfiedLinkError...

Error occurred during initialization of boot layer
java.lang.UnsatisfiedLinkError: 'int sun.nio.fs.UnixNativeDispatcher.init()'

looks like the set of enabled modules is the cause of the problem! With:

diff --git build.sbt build.sbt
diff --git build.sbt build.sbt
index bbdeef42ce..00f114fa64 100644
$ git diff
diff --git build.sbt build.sbt
diff --git build.sbt build.sbt
index bbdeef42ce..764f9bf079 100644
--- build.sbt
+++ build.sbt
@@ -3912,7 +3912,7 @@ lazy val `engine-runner` = project
       val NI_MODULES =
         "org.graalvm.nativeimage,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.base,org.graalvm.nativeimage.driver,org.graalvm.nativeimage.librarysupport,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto,com.oracle.graal.graal_enterprise,com.oracle.svm.svm_enterprise"
       val JDK_MODULES =
-        "java.desktop,java.naming,java.net.http,jdk.charsets,jdk.crypto.ec,jdk.localedata,jdk.httpserver,java.rmi"
+        "java.naming,java.net.http,jdk.charsets,jdk.crypto.ec,jdk.localedata,jdk.httpserver,java.rmi"
       val DEBUG_MODULES  = "jdk.jdwp.agent"
       val PYTHON_MODULES = "jdk.security.auth,java.naming"

it is possible to execute

enso$ ./built-distribution/enso-engine-*/enso-*/bin/enso --jvm --run test/Base_Tests

after building native image executable with ENSO_LAUNCHER=-ls,fast,debug.

[JaroslavTulach]

• This may be the core cause of [Native Image] Dynamic loading of jvm.dlldoesn't link - regression since GraalVM21? oracle/graal#11152
• the jvm.lib or jvm.obj then appears multiple times in the linkage

[JaroslavTulach]

• Let's see if it works in da05d8d!?
• Use the generated executable JaroslavTulach/LaunchJdkViaJni#3 indicates it could work

[JaroslavTulach]

• the commit 967e2ce starts with following idea
• define poi-wrapper fat JAR
• use that poi-wrapper in std-table instead of direct dependecy on poi
• make sure the poi-wrapper contains the necessary patches to avoid dependency on java.desktop
• I was hoping to find inspiration at Updating snakeyaml and removing no longer needed patches #11207 - but it doesn't define reusable fat JAR either!

[JaroslavTulach]

• done by @Akirathan at 179aff3
• thank you - it's a complicated change - I could never do it myself
• but the CI is green!
• finally we can move pass this JNI struggle...

[JaroslavTulach]

• Patches to avoid dependency on java.desktop by Apache POI
• proposal sent to upstream project
  • Avoiding needless java.awt.Color dependency in HSSFColor apache/poi#806
• once it is integrated
• let's switch to new POI version and remove this copy of HSSFColor!

[Akirathan]

So if I understand it correctly, all the Standard Library Tests (jvm) use the new launch-in-the-same-process technique?

[JaroslavTulach]

This change is already present in this Nightly Release:

• thanks to Check NI size after it is built #12996 we can check the current sizes
• Linux 404MB
• MacOSX Arm 392MB
• MacOSX Intel 377MB
• Windows 390MB

These values should be smaller, because we removed java.desktop dependency from native image.