Use of different JDKs for development vs production

[GregDritschler]

Summary

I ran into an interesting problem when using the java-microprofile stack. My application uses a database. It worked perfectly fine when running the development image on my laptop. When I used the production image, it completely broke. Every database request failed.

[ERROR ] CWWJP9992E: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.7.4.v20190115-ad5b7c6b2a): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: SSL error: Received fatal alert: handshake_failure DSRA0010E: SQL State = 08006, Error Code = 0
Error Code: 0

The problem was due to a difference in the SSL implementation in the OpenJDK used by the development image vs. the IBM JDK used by the production image. Specifically the IBM JDK requires a property to be set to match the behavior of other JDKs when getting an SSLContext.

https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/matchsslcontext_tls.html

Luckily I spotted the difference in JDK implementations immediately and suspected that was the issue. Someone else who is not experienced may not be aware of the role the JDK plays in this and might be puzzled by this and spend a lot of time trying to solve the issue.

In fact my first reaction (and the reaction of someone else on my team) was "it works on my machine, why isn't it working on our cluster [where we were running the production image]?"

I think the use of different JDKs for development and production violates a basic principal of container-based development. To quote from the 12-factor app document

"A twelve-factor app never relies on implicit existence of system-wide packages. It declares all dependencies, completely and exactly, via a dependency declaration manifest. Furthermore, it uses a dependency isolation tool during execution to ensure that no implicit dependencies “leak in” from the surrounding system. The full and explicit dependency specification is applied uniformly to both production and development."

https://12factor.net/dependencies

[ykoyfman]

Can confirm, I ran into this issue also.

[stephenkinder]

@GregDritschler are you saying the container you used to develop the application was different than the container that was deployed in Kabanero? Typically, Champ (EA) is making available the same container that Jane (dev) should use in development.

[GregDritschler]

When using kabanero's java-microprofile stack, you get a different JDK when running the container built by appsody run vs the container built by appsody build.

Output from the appsody run container:

[Container] [INFO] Launching defaultServer (Open Liberty 19.0.0.9/wlp-1.0.32.cl190920190905-0148) on Eclipse OpenJ9 VM, version 1.8.0_222-b10 (en_US)

Output from the appsody build container:

Launching defaultServer (Open Liberty 19.0.0.9/wlp-1.0.32.cl190920190905-0148) on IBM J9 VM, version 8.0.5.40 - pxa6480sr5fp40-20190807_01(SR5 FP40) (en_US)

[groeges]

@yeekangc When we modified the java-microprofile stack to use UBI images needed for Kabanero, we used the docker images from openliberty/open-liberty.

The latest version of the images have a image that uses OpenJDK-OpenJ9 in the UBI images but the 19.0.0.9 images don't seem to have OpenJDK-OpenJ9 versions, but all seem to use IBM J9.

This then causes differences between the Appsody images (which use OpenJDK-OpenJ9 for both the run and build images) and also the Kabanero run image ( which use OpenJDK-OpenJ9).

If there is a 19.0.0.9 version of the UBI image that uses OpenJDK-OpenJ9 we could just use that but currently I cant find one. Is there one?

If there is not a 19.0.0.9 version of the UBI image that uses OpenJDK-OpenJ9 then a different solution is required to be able to get the run and build images to use compatible dependant packages etc.

[scottkurz]

This is a much better story in the newer versions of the java-microprofile stack (e.g. the latest now, v0.2.25). It is also basically fixed in the new java-openliberty stack.

In my local build of this stack, the stack image uses:

OpenJDK Runtime Environment (build 1.8.0_222-b10)
Eclipse OpenJ9 VM (build openj9-0.15.1, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20190717_368 (JIT enabled, AOT enabled)

with the app build image using:

openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
Eclipse OpenJ9 VM (build openj9-0.17.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191017_442 (JIT enabled, AOT enabled)

Note the fix level is still different, however.... (1.8.0 both but 222 vs 232).

I think it's fair to say it's up to each stack author to decide what level of fidelity to enforce here, and the java-microprofile and java-openliberty stacks now, out of the box, have at least the level of alignment that we've intended so far.

Opinions welcome going forward, but at least the very big discrepancy of IBM vs Open J9 was fixed here.