Added GraalVM native support to Apache FreeMarker

[fugerit79]

Native support

In this pull request :

GraalVM native support

1. Added native-image.properties to initialize at runtime relevant classes
2. Added resource-config.json to track resources to include in the native executable
3. Added boolean IS_GRAALVM_NATIVE = System.getProperty( "org.graalvm.nativeimage.imagecode" ) != null in logger to avoid runtime initialization error
4. Added dependency to GraalVM SDK

Test module

Added a test module to build as a GraalVM native image.

1. Build the main Apache FreeMarker proejct :

./gradlew build -Pfreemarker.allowUnsignedReleaseBuild=true

2. Build the test module native image with GraalVM :

./gradlew :freemarker-test-graalvm-native:nativeCompile 

3. Run the project :

./freemarker-test-graalvm-native/build/native/nativeCompile/freemarker-test-graalvm-native 

CI

Updated CI to test the GraalVM native support by building and running the test module as a native executable.

[fugerit79]

@ddekany

I misunderstood at the beginning.

I’ve pushed some modifications, and hopefully, it should be fine now.

I’ve also done some additional testing and made the following changes:

• Improved the native test by adding relevant reflection configurations.
• Added some comments.
• Fixed a couple of typos.
• Substitution (Log4jOverSLF4JTesterSubstitute) is no more needed after graalvm native check in logger

Additionally I tested the pull request on my fork : 

https://github.com/fugerit-org/freemarker/actions/workflows/ci.yml

I really hope now everything is ok.

See you

[ddekany]

But that still prefers Log4J API instead of SLF4J API when we have log4j-over-slf4j, under GrallVM Native only, and it should prefer SLF4J in that case. I know, this Logger class is a mess. Anyway, I looked into more, and the easiest way to achieve what I said is just changing isAutoDetected so that if IS_GRAALVM_NATIVE is true, then we return true for LIBRARY_SLF4J. Actually, even better long term, let's return true for LIBRARY_COMMONS too, and that way for GraalVM Native we just brought ahead what's planned for 2.4 anyway.

[fugerit79]

I'm having a deeper look at it.

[fugerit79]

@ddekany let's see if this is ok for you.

I preferred to create separate methods so maybe it's easier to understand than complex conditional statements, do you agree?

    // legacy auto-detection (until FreeMarker 2.3.X)
    private static boolean isAutoDetectedLegacy( int libraryEnum ) {
        return !(libraryEnum == LIBRARY_AUTO || libraryEnum == LIBRARY_NONE
                || libraryEnum == LIBRARY_SLF4J || libraryEnum == LIBRARY_COMMONS);
    }

    // next generation auto-detection (FreeMarker 2.4.X and on)
    private static boolean isAutoDetectedNG( int libraryEnum ) {
        return !(libraryEnum == LIBRARY_AUTO || libraryEnum == LIBRARY_NONE);
    }

    private static boolean isAutoDetected(int libraryEnum) {
        // 2.4: Remove libraryEnum == LIBRARY_SLF4J || libraryEnum == LIBRARY_COMMONS (use only isAutoDetectedNG()=
        return IS_GRAALVM_NATIVE ? isAutoDetectedNG(libraryEnum) : isAutoDetectedLegacy(libraryEnum);
    }

At beginning didn't fully grasped the meaning of all logger detection logic. I hope that now it is ok.

[ddekany]

Don't we also need to add freemarker/ext/beans/unsafeMethods.properties accessed from freemarker.ext.beans.LegacyDefaultMemberAccessPolicy here?

[fugerit79]

Maybe 'freemarker/ext/beans/unsafeMethods.properties' when freemarker.ext.beans.LegacyDefaultMemberAccessPolicy is reachable? (just commited it this way)

do you think is it better to squash commits?

[ddekany]

new Version(Configuration.getVersion().toString()), because, recycling the current version like that is in principle not allowed (current only logs a WARN, eventually will throw exception). (Because, people almost always do this by mistake, not understanding what that setting does. So in the rare cases where you really mean to do it, there's some gymnastic involved.)

[fugerit79]

Ok, updated this one too. (in production projects I never resuse it).

[ddekany]

Please add the licence header to the template instead (inside <#-- ... -->)

[fugerit79]

I think this is done too. Sorry If i missed it

[fugerit79]

@ddekanyI have squashed everything into two commits :

• GraalVM native support implementation
• Tests and CI

I hope all your requests have been fulfilled.

Regards

[klopfdreh]

Hey,

thanks a lot for all the effort already invested here. 👍

I think it would be very useful to include a hint in the README.md to let users of Freemarker know that it is important to register their models for reflections (reflection-config.json), with some minor notes how to place them (link to GraalVM documentation)

[fugerit79]

You are right @klopfdreh,

I’ve tried to do my best with this addition—let me know if it looks good to you:
9c8e84c

Thanks in advance!
Matteo

P.S.: Initially, I was considering it, but since the README file was quite slim, I thought it would be better to include this kind of information in the broader documentation later on.

[OnnoH]

Following this thread with interest @fugerit79. Thanks for the work you put in! I'm developing a PicoCLI app and would like to include Freemarker. Sadly I'm unable to build it (because of Apple Silicon?)

Regarding documentation: Next to the GraalVM reference, an example reflection entry might help clarifying things.

[ddekany]

What's the build error? You are unable to build this branch, or FreeMarker in general?

[fugerit79]

Hello @OnnoH

Which is the build issue you are experiencing? (can you provide a reproducer and explain what is not working exactly?)
Keep in mind to build freemakrer you need gradle to be able to find jdj 8, 16 and 17.

@ddekany @OnnoH The CI build is working smoothly on my fork :
https://github.com/fugerit-org/freemarker/actions/runs/14956395790/job/42012623892
For windows, ubuntu amd64 and ubuntu arm.

I've tried a local build with my macboc pro M1 and that worked too.

About the reflecting sample I linked the reachability metadata repository and the documentation, where there are plenty of examples.

[ddekany]

@OnnoH Here's a build of this branch: https://freemarker.apache.org/builds/pr121/

[OnnoH]

Thanks @fugerit79 and @ddekany. It was a toolchain issue and after installing the required legacy JDKs, the build worked! (After migrating to a new machine, I never thought about installing them ;-)

Adding the new .jar to my project it produced the expected results when passing in an object or an object in a map. The picocli-codegen annotation processor added the needed reflection for this class automatically.

When starting my application, it spits out this error:
ERROR freemarker.log.LoggerFactory: Unexpected error when initializing logging for "SLF4J". Exception: java.lang.RuntimeException: Unexpected error when creating logger factory for "SLF4J". Caused by: java.lang.ClassNotFoundException: freemarker.log._SLF4JLoggerFactory

Do I need to include the Slf4J dependencies in my project?

[fugerit79]

Hello @OnnoH , no you do not need add SLF4J if you do not use it already.

I suspect for some reason this property is not set : "org.graalvm.nativeimage.imagecode" (I tested it in many scenarios and it is supposed to be always set at runtime on a GraalVM generated executable).

Can you provide :

• runtime value of property "org.graalvm.nativeimage.imagecode"? For instance print at application startSystem.out.println( "org.graalvm.nativeimage.imagecode : "+System.getProperty( "org.graalvm.nativeimage.imagecode" ) )
• build info (gradle, maven and especially GraalVM version).
• full stack trace

Thanks in advance.

[OnnoH]

Sure @fugerit79.

Image code: org.graalvm.nativeimage.imagecode : runtime
Maven build: native-maven-plugin
GraalVM version: OpenJDK Runtime Environment GraalVM CE 23.0.2+7.1 (build 23.0.2+7-jvmci-b01)
Stacktrace: not available only the error
ERROR freemarker.log.LoggerFactory: Unexpected error when initializing logging for "SLF4J". Exception: java.lang.RuntimeException: Unexpected error when creating logger factory for "SLF4J". Caused by: java.lang.ClassNotFoundException: freemarker.log._SLF4JLoggerFactory
when initialising a Configuration
Configuration cfg = new Configuration(Configuration.VERSION_2_3_34);

[fugerit79]

Hello @OnnoH it seems to me the configuration is ok.

Are you using maven as build system?

Are you sure your application has been actually built with your FreeMarker build?

I tested the build on a few maven applications, but to do so I did :

1. Publish freemarker gae artifact to maven local repository ad the end of the gradle buildpublishToMavenLocal :

./gradlew "-Pfreemarker.signMethod=none" "-Pfreemarker.allowUnsignedReleaseBuild=true" --continue clean build publishToMavenLocal

2. Add dependency for :

    <dependency>
      <groupId>org.freemarker</groupId>
      <artifactId>freemarker-gae</artifactId>
      <version>2.3.35-SNAPSHOT</version>
    </dependency>

3. Remove dependency for every artificat which needs it :

    <dependency>
      <groupId>org.fugerit.java</groupId>
      <artifactId>fj-doc-base</artifactId>
      <exclusions>
        <exclusion>
          <groupId>org.freemarker</groupId>
          <artifactId>freemarker</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

Here you can find an example.

You can check if the "standard" FreeMarker jar is still included in your build by running :

mvn dependency:tree

If you want you can provide the output. The result should only include freemarker-gae artifact.

Note : That is needed just because this pull request has not yet merged and published in an official release of FreeMarker.

[ddekany]

BTW, you can also have an ${.version} in a template, and see if it prints 2.3.35-SNAPSHOT.

[OnnoH]

Yes I use Maven for my project @fugerit79 and the new .jar was the only freemarker entry. Just to be sure, I ran the commands you suggested.
BUILD SUCCESSFUL in 45s
68 actionable tasks: 64 executed, 4 up-to-date

mvn dependency:tree | grep freemarker
[INFO] +- org.freemarker:freemarker-gae:jar:2.3.35-SNAPSHOT:compile

Including ${.version} in my template @ddekany gives:
<version>2.3.35-nightly</version>
as does Configuration.getVersion()

[fugerit79]

@OnnoH, bug if you can handle and print version in template, it means after the error :

_"ERROR freemarker.log.LoggerFactory: Unexpected error when initializing logging for "SLF4J". Exception: java.lang.RuntimeException: Unexpected error when creating logger factory for "SLF4J". Caused by: java.lang.ClassNotFoundException: freemarker.log.SLF4JLoggerFactory"

The application is working? or you printed the version with a non native build?

Thanks in advance.

[ddekany]

@OnnoH Yeah, it actually prints "nightly", not SNAPSHOT... same thing. Certainly I'm stating the obvious, but you also have to be sure that you have built that from the branch of the PR.

[fugerit79]

Thanks @ddekany ,

@OnnoH just to be completely sure, are you building the project from the branch on my fork? (as the PR has not been merged yet) :

https://github.com/fugerit-org/freemarker/tree/1-add-graalvm-support-to-apache-freemarker

[OnnoH]

Yes, I'm sure @fugerit79 .

> git remote show origin
* remote origin
  Fetch URL: git@github.com:fugerit-org/freemarker.git
  Push  URL: git@github.com:fugerit-org/freemarker.git
  HEAD branch: 2.3-gae
  Remote branches:
    1-add-graalvm-support-to-apache-freemarker tracked
    1-freemarker-test-graalvm-native           tracked
    1-poc                                      tracked
    2.3-gae                                    tracked
    branch-sonar-cloud                         tracked
  Local branches configured for 'git pull':
    1-add-graalvm-support-to-apache-freemarker merges with remote 1-add-graalvm-support-to-apache-freemarker
    2.3-gae                                    merges with remote 2.3-gae
  Local refs configured for 'git push':
    1-add-graalvm-support-to-apache-freemarker pushes to 1-add-graalvm-support-to-apache-freemarker (up to date)
    2.3-gae                                    pushes to 2.3-gae                                    (up to date)

Indeed after the error, the application works as expected.

[fugerit79]

@OnnoH ok I see. Thanks for the feedback.

I'd like to check it anyway.

any chance you have a reproducer code? (Maybe a public repository?)

[OnnoH]

@fugerit79

any chance you have a reproducer code? (Maybe a public repository?)

Good news. I did start a project from scratch and the error no longer shows up!

I'll peel back the layers of my other project to see what might cause the error. If I can reproduce it, I'll let you know. But the project is a bit of a mess because I dug too many rabbit holes, so it might take some time 😁

[OnnoH]

@fugerit79 @ddekany

Found the culprits!

        <dependency>
            <groupId>com.spotify</groupId>
            <artifactId>github-client</artifactId>
            <version>${github-client.version}</version>
        </dependency>

and

        <dependency>
            <groupId>org.eclipse.jgit</groupId>
            <artifactId>org.eclipse.jgit</artifactId>
            <version>${jgit.version}</version>
        </dependency>

They both have a transitive dependency to Slf4J.

Although never called, the mere presence of

final GitHubClient githubClient = GitHubClient.create(URI.create(API_GITHUB_URL), API_GITHUB_TOKEN);

was apparently enough to throw the error.

I'm will add it to the sample project and share it here when ready.

[ddekany]

If the org.slf4j.Logger class exists, then FreeMarker tries to use SLF4J for its own logging. That then it fails with java.lang.ClassNotFoundException: freemarker.log._SLF4JLoggerFactory is a problem. I guess (not sure), since the LoggerFactory implementations are only instantiated via Java reflection, they should be added to native-image.properties.

[fugerit79]

@ddekany, yes, actually it should be added to resources/META-INF/native-image/org.freemarker/freemarker/reflect-config.json.
I'm testing it before commit.

[fugerit79]

@ddekany @OnnoH

I added reflection configuration for LoggerFactory implementations :

fugerit-org@c28e703

I was able to reproduce and test the behaviour on a POC repository :

https://github.com/fugerit-org/poc-freemarker-graalvm

I tested :

1. Java Util Logging on branch vanilla-jul
2. SLF4J simple on branch slf4j-simple

It seems to work now.

Note 1 : I previously tested mainly on modern framework like Quarkus or SpringBoot, which have a good built in support for GraalVM. Now the PR should be more stable.

[OnnoH]

Thanks for all the work @fugerit79 and @ddekany!

I pulled the changes, ran a build and tested it with my pet project and this sample: https://github.com/OnnoH/picocli-freemarker

And it's squeaky clean 😜

[fugerit79]

@OnnoH thanks to you for the debugging :)

[klopfdreh]

Just one side note - if you are using Spring Boot you also can use RuntimeHintsRegistrar

https://github.com/klopfdreh/github-api-native-test/blob/main/src/main/java/github/api/nat/test/aot/GitHubRuntimeHints.java

During the AOT processing at build time you can create the hints dynamically.

[ddekany]

@fugerit79 Thanks for the fixes! Regarding the README (discussed much earlier), feel free to add any pointer to it, based on the discussion here. After the merge (in a few days I think) I will move that information over into the documentation, and leave a pointer to that in the README, so no worries about the length.

[fugerit79]

@ddekany Of course, is it ok this guide?

fugerit-org@86dc986

I created a sample project to create this guide, if you want you can quote it :

https://github.com/fugerit-org/freemarker-graalvm-sample/tree/main/native-image

If you think it is useful I can add a short guide for Gradle and Maven.

UPDATE : I added Maven :

https://github.com/fugerit-org/freemarker-graalvm-sample/tree/main/native-image-maven

and Gradle KTS sample :

https://github.com/fugerit-org/freemarker-graalvm-sample/tree/main/native-image-gradle

And a few more for quarkus, micronaut, springboot and helidon :

https://github.com/fugerit-org/freemarker-graalvm-sample/blob/main/README.md

[fugerit79]

@ddekany I did a rebase on 2.3-gae just in case.