Java 11 - Cannot access java.sql classes with isolated classloader

[MrWalterWhite2]

Hi,
For some reason unknown, java.sql classes are not found when using Java 11 with isolated classloader, a java.lang.NoClassDefFoundError exception is raised in mirth.log, even when we change the mcservice.vmoptions to include the recommended JAVA9+ entries.

Here is a test package I've created and compiled with Java 11:

package test;

import java.sql.SQLException;

public class NeedsJavaSql {
  public static void testSQLException() {
    SQLException e = new SQLException("Testing access to java.sql.SQLException");
  }
}

In Mirth Connect, I've crated a custom folder where I copied my test JAR file and added it as Channel Resource to a test channel, with only a JavaScript Reader where I've put the following code:

var iso = Packages(contextFactory.getIsolatedClassLoader());
iso.test.NeedsJavaSql.testSQLException();

In mirth.log this is the error that is generated:

ERROR 2022-03-06 22:22:07,883 [JavaScript Reader Polling Thread on TEST_JAVASQL_CHANNEL (eb1e4255-8c27-44c4-9d37-167fffa30f96) < eb1e4255-8c27-44c4-9d37-167fffa30f96_Worker-1] com.mirth.connect.connectors.js.JavaScriptReceiver: Error executing JavaScript Reader script 649b1e92-5739-4d05-aacf-fb006e708542.
com.mirth.connect.server.util.javascript.JavaScriptExecutorException: java.lang.NoClassDefFoundError: java/sql/SQLException
	at com.mirth.connect.server.util.javascript.JavaScriptUtil.execute(JavaScriptUtil.java:75)
	at com.mirth.connect.connectors.js.JavaScriptReceiver.poll(JavaScriptReceiver.java:101)
	at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.NoClassDefFoundError: java/sql/SQLException
	at test.NeedsJavaSql.testSQLException(NeedsJavaSql.java:7)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
	at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:226)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1669)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:1009)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:412)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3545)
	at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:121)
	at com.mirth.connect.server.util.javascript.JavaScriptTask.executeScript(JavaScriptTask.java:150)
	at com.mirth.connect.server.util.javascript.JavaScriptUtil.executeScript(JavaScriptUtil.java:542)
	at com.mirth.connect.connectors.js.JavaScriptReceiver$JavaScriptReceiverTask.doCall(JavaScriptReceiver.java:159)
	at com.mirth.connect.server.util.javascript.JavaScriptTask.call(JavaScriptTask.java:113)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: java.sql.SQLException
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 21 more

If I deploy my test JAR file in "custom-lib" folder (without using isolated classloader) with Java 11, there is no error, so I assume access to java.sql is ok:
var obj = Packages.test.NeedsJavaSql.testSQLException();

With Java 8, both situations work without problems (with "custom-lib" and isolated classloader).

In summary:

• with Java 8: access to java.sql is working when JAR is deployed in both "custom-lib" and with an isolated classloader;
• with Java 11: access to java.sql is working when JAR is deployed in "custom-lib" but fails when used with an isolated classloader;

There are already two reported bugs also pointing to problems with accessing java.sql classes in Java 11:
[BUG] Third-party JDBC drivers and java.sql.driver #4719
[BUG] DatabaseConnectionFactory.createDatabaseConnection with a custom driver fails on Java9+ #4587

Am I missing something?

Thanks

[jonbartels]

Good writeup! I can see and understand your issue. I am surprised that is happening.

Do your channel dependencies depend on ONLY your resource or your resource and the "Default Resource"?

Using custom-lib is a workaround but you should be able to do all this from a real resource directory.

Does your use case require an isolated classloader? Does your JAR and test case work with the standard classloader?

[MrWalterWhite2]

Hi jonbartels

Do your channel dependencies depend on ONLY your resource or your resource and the "Default Resource"?

No, just depends on my "channel-lib" resource directory. In fact, my default "custom-lib" resource directory is empty.

Using custom-lib is a workaround but you should be able to do all this from a real resource directory.

Yes, that was my expectation, at least that's how it is working with Java 8.

Does your use case require an isolated classloader? Does your JAR and test case work with the standard classloader?

Yes, my final use case depends on newer versions of already shipped JAR libraries along with Mirth Connect, so I have read somewhere that the best approach was to deploy all JAR files in a channel specific resource directory and use an isolated classloader.
Since I was using Java 11 to develop my JAR library and run Mirth, I started seeing this "java.lang.ClassNotFoundException: java.sql.SQLException" errors in mirth.log that I could not understand. Compiling it with Java 8 and switching Mirth to use it too, just fix the issue but I need to understand why is this happening.

[jonbartels]

Using resource directories is best. Adding things to custom-lib is not officially deprecated but IMO it should be discouraged.

Try your use case again by depending on both "Default Resource" and your own channel-lib resource. I cannot cite sources but I seem to recall not having "Default Resource" causing a similar issue for me in the past.

You are correct that using the isolated classloader is the right process for using newer JARs than what ships with MC.

[tonygermano]

I believe this is another use case where this ticket would avoid the issue #4075

[MrWalterWhite2]

Hi jonbartels,

Try your use case again by depending on both "Default Resource" and your own channel-lib resource. I cannot cite sources but I seem to recall not having "Default Resource" causing a similar issue for me in the past.

Thanks for the suggestion but did not worked either.

This is definitely a Java 9+ problem because I've recompiled it with Java 8 and changed back Mirth Connect to run also with Java 8 and it is working without problems.

[jonbartels]

I see that mcservice-java9+.vmoptions has --add-modules=java.sql.rowset I wonder if the entire java.sql package needs to be defined.

[ChristopherSchultz]

This probably has more to do with hoe the isolated classloader is being created than anything else.

Here is an important reference; read the final comment when this bug report was closed as "not an issue":
https://bugs.openjdk.java.net/browse/JDK-8161269

The issue is that in Java 9 and later, the behavior of a ClassLoader whose parent ClassLoader is null is different than that of Java 8 and earlier. Most people would probably expect that a null ClassLoader would end up giving access to everything available to the JVM like basically java.* as before. But not so.

In Java 9 and later, you get only the real core packages like java.lang. But definitely not java.sql which requires an --add-module upon JVM launch.

The isolated ClassLoader is created here:
https://github.com/nextgenhealthcare/connect/blob/development/server/src/com/mirth/connect/server/util/javascript/MirthContextFactory.java#L107

... and I think the second argument to the URLClassLoader constructor should probably be either ClassLoader.getPlatformClassLoader() or ClassLoader.getSystemClassLoader() but I don't know enough about the differences to know which one would work the way you "expect" it to work.

Unfortunately, those methods are only available in Java 9 and later, and I believe Mirth needs to be complied (or at least compilable) using Java 8, so it would need to be done using reflection just in case those methods aren't available at runtime.

[MrWalterWhite2]

Hi ChristopherSchultz,
Your explanation makes totally sense to me.
Let's hope the behavior of MC could change in future versions on how the isolated classloader is created, so that more options could exist to load specific libraries per channel without loosing access to base Java functionality.
Thx.

[jbeckers]

I ran into this today running the 4.5.2 docker image (Java 17) :

java.lang.NoClassDefFoundError: java/sql/Driver
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:524) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:427) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:421) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:420) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:592) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?]
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:524) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:427) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:421) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:420) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:592) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?]
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:524) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:427) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:421) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:420) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:592) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?]
        at com.mirth.connect.connectors.jdbc.CustomDriver.<init>(CustomDriver.java:40) ~[jdbc-server.jar:?]
        at com.mirth.connect.server.userutil.DatabaseConnectionFactory.initializeDriverAndGetInfo(DatabaseConnectionFactory.java:146) ~[mirth-server.jar:?]
        at com.mirth.connect.server.userutil.DatabaseConnectionFactory.getCustomDriverInfo(DatabaseConnectionFactory.java:168) ~[mirth-server.jar:?]
        at com.mirth.connect.server.userutil.DatabaseConnectionFactory.createDatabaseConnection(DatabaseConnectionFactory.java:55) ~[mirth-server.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138) ~[rhino-1.7.13.jar:1.7.13]
        at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:226) ~[rhino-1.7.13.jar:1.7.13]
        at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1692) ~[rhino-1.7.13.jar:1.7.13]
        at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:1013) ~[rhino-1.7.13.jar:1.7.13]
        at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) ~[rhino-1.7.13.jar:1.7.13]
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:412) ~[rhino-1.7.13.jar:1.7.13]
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3578) ~[rhino-1.7.13.jar:1.7.13]
        at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:121) ~[rhino-1.7.13.jar:1.7.13]
        at com.mirth.connect.server.util.javascript.JavaScriptTask.executeScript(JavaScriptTask.java:151) ~[mirth-server.jar:?]
        at com.mirth.connect.connectors.js.JavaScriptDispatcher$JavaScriptDispatcherTask.doCall(JavaScriptDispatcher.java:261) ~[js-server.jar:?]
        at com.mirth.connect.connectors.js.JavaScriptDispatcher$JavaScriptDispatcherTask.doCall(JavaScriptDispatcher.java:221) ~[js-server.jar:?]
        at com.mirth.connect.server.util.javascript.JavaScriptTask.call(JavaScriptTask.java:114) ~[mirth-server.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.lang.ClassNotFoundException: java.sql.Driver
        at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:592) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?]
        ... 54 more

Your analysis is correct.

I was intrigued by the comment "In Java 9 and later, you get only the real core packages like java.lang. But definitely not java.sql which requires an --add-module upon JVM launch."
It seemed unlikely that would change anything. I added --add-modules=java.sql to the mcserver vmoptions, and it indeed does not fix the issue.

I think the only way around this issue, for now, is to sidestep the isolated classloader, by putting libs in server-lib instead of custom-lib.

[dfauraz]

PR created: #5470

[jonbartels]

@MrWalterWhite2 - NextGen pushed a commit for a child-first classloader this week ce589cf

This will resolve #5644 which is a duplicate of various other classloader cases.

Tagging @dfauraz - Since you sent in a PR I thought this commit might be interesting to you.