Check NI size after it is built #12996

Akirathan · 2025-04-30T10:59:26Z

Pull Request Description

Size of the generated enso NI executable is checked right after it was built. The size bounds for the production NI are set to 150 MB <= x <= 450 MB, explained in #12996 (comment).

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

Akirathan · 2025-04-30T11:59:23Z

The test is currently failing - size should be 150MB < size < 450MB, but is 489MB. The reason for that is that the NI built for the test is built with ENSO_LAUNCHER=native,test, which means that it is not optimized for size. That is done for NI build for the production.

Two possible solutions for that:

Move this test to nightly checks, that build NI for production.
Increase allowed size to 500 MB and leave the test here.

I personally prefer solution (1).

This reverts commit cb94968.

This reverts commit 48bf3cf.

Akirathan · 2025-05-05T16:02:22Z

In c324144, the check is implemented directly in sbt, during buildEngineDistribution.

Example of NI build failing due to unexpected size:

$ env ENSO_LAUNCHER=native,-ls,fast sbt buildEngineDistribution
...
[info] ------------------------------------------------------------------------------------------------------------------------
[info]                         8.3s (6.0% of total time) in 665 GCs | Peak RSS: 8.28GB | CPU load: 9.28
[info] ------------------------------------------------------------------------------------------------------------------------
[info] Build artifacts:
[info]  /home/pavel/dev/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso (executable)
[info] ========================================================================================================================
[info] Finished generating 'enso' in 2m 17s.
[info] built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso native image build successful.
[error] Generated binary built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso has unexpected size: 151 MB. Expected size is between 200 and 550 MB.
[error] java.lang.RuntimeException: Generated binary built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso is too large
[error] 	at NativeImage$.$anonfun$checkNativeImageSize$1(NativeImage.scala:349)
[error] 	at NativeImage$.$anonfun$checkNativeImageSize$1$adapted(NativeImage.scala:326)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
[error] 	at java.base/java.lang.Thread.run(Thread.java:1447)
[error] (engine-runner / checkNativeImageSize) Generated binary built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso is too large

Successful build of NI:

$ env ENSO_LAUNCHER=native,-ls,fast sbt buildEngineDistribution
...
[info] Build artifacts:
[info]  /home/pavel/dev/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso (executable)
[info] ========================================================================================================================
[info] Finished generating 'enso' in 2m 25s.
[info] built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso native image build successful.
[info] Generated binary built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso is within the expected size: 151 MB.
[info] Generating index for built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/AWS 
[info] Generating index for built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Base 
[info] Generating index for built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database 
[info] Generating index for built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Examples 
[info] Generating index for built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Geo 
...

Akirathan · 2025-05-05T16:09:30Z

project/NativeImage.scala

+      )
+    }
+    val bytes              = generatedBin.attributes.size()
+    val productionNIBounds = (150, 450)


Based on latest nightly build:

Linux production NI has 433 MB

Windows production NI has 390 MB

MacOS aarch64 production NI has 393 MB

MacOS x64 production NI has 378 MB

Maybe the maxImageSizeInMb function should return different value for different OS to allow incremental individual improvements on each platform

JaroslavTulach

Put it in, so the size never regresses (too much) again!

JaroslavTulach · 2025-05-06T07:59:41Z

project/NativeImage.scala

+        "Ensure that the dependency on `buildNativeImage` is properly set."
+      )
+    }
+    val bytes              = generatedBin.attributes.size()


the constant should be more prominent

def GraalVM.EnsoLauncher.maxImageSizeInMb = 450 maybe?

it shouldn't be hidden here

JaroslavTulach · 2025-05-06T08:00:45Z

project/NativeImage.scala

+      )
+    }
+    val bytes              = generatedBin.attributes.size()
+    val productionNIBounds = (150, 450)


Maybe the maxImageSizeInMb function should return different value for different OS to allow incremental individual improvements on each platform

Akirathan · 2025-05-06T10:47:14Z

With ee07a3c, the expected sizes of the native image are defined in GraalVM.NativeImageSize:

enso/project/GraalVM.scala

Lines 99 to 106 in ee07a3c

    
           // Expected production NI sizes deduced from sizes on latest 
        
           // nightly builds: https://github.com/enso-org/enso/pull/12996#discussion_r2073742680 
        
           // With maximal size relaxed by 30 MB. 
        
           private val windowsX64Release = NativeImageSize(200, 420) 
        
           private val linuxX64Release   = NativeImageSize(200, 463) 
        
           private val macX64Release     = NativeImageSize(200, 408) 
        
           private val macARM64Release   = NativeImageSize(200, 423) 
        
           private val testNISize        = NativeImageSize(100, 550)

cc @JaroslavTulach

JaroslavTulach · 2025-05-11T14:25:05Z

As of May 10, 2025 the sizes are: Linux 404MB, MacOSX Arm 392MB, MacOSX Intel 377MB, Windows 390MB

* Add Enso test for NI size * better test clue * Revert "better test clue" This reverts commit cb94968. * Revert "Add Enso test for NI size" This reverts commit 48bf3cf. * NI size is checked in sbt during buildEngineDistribution * NativeImageSize constants are more visible, and platform specific

Add Enso test for NI size

48bf3cf

Akirathan added the CI: No changelog needed Do not require a changelog entry for this PR. label Apr 30, 2025

Akirathan self-assigned this Apr 30, 2025

Akirathan requested review from jdunkerley, radeusgd, GregoryTravis, AdRiley and marthasharkey as code owners April 30, 2025 10:59

better test clue

cb94968

enso-bot bot mentioned this pull request Apr 30, 2025

Report unused imports #12825

Closed

GregoryTravis approved these changes Apr 30, 2025

View reviewed changes

Akirathan added 2 commits May 5, 2025 17:03

Revert "better test clue"

8d77f79

This reverts commit cb94968.

Revert "Add Enso test for NI size"

18d496f

This reverts commit 48bf3cf.

NI size is checked in sbt during buildEngineDistribution

c324144

Akirathan requested review from 4e6, JaroslavTulach and hubertp as code owners May 5, 2025 16:03

Akirathan commented May 5, 2025

View reviewed changes

Akirathan changed the title ~~Add unit test for NI size~~ Check NI size after it is built May 5, 2025

JaroslavTulach approved these changes May 6, 2025

View reviewed changes

NativeImageSize constants are more visible, and platform specific

ee07a3c

Akirathan merged commit cc60c25 into develop May 6, 2025
79 of 81 checks passed

Akirathan deleted the wip/akirathan/ni-size-unit-test branch May 6, 2025 16:55

JaroslavTulach mentioned this pull request May 11, 2025

Handle --jvm flag in process with JNI instantiated JVM #12843

Merged

2 tasks

enso-bot bot mentioned this pull request May 12, 2025

Callback to native executable from HotSpot JVM #13076

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Check NI size after it is built #12996

Check NI size after it is built #12996

Uh oh!

Akirathan commented Apr 30, 2025 •

edited

Loading

Uh oh!

Akirathan commented Apr 30, 2025 •

edited

Loading

Uh oh!

Akirathan commented May 5, 2025 •

edited

Loading

Uh oh!

Akirathan May 5, 2025

Uh oh!

JaroslavTulach May 6, 2025

Uh oh!

JaroslavTulach left a comment

Uh oh!

JaroslavTulach May 6, 2025

Uh oh!

JaroslavTulach May 6, 2025

Uh oh!

Akirathan commented May 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

JaroslavTulach commented May 11, 2025

Uh oh!

Uh oh!

Check NI size after it is built #12996

Check NI size after it is built #12996

Uh oh!

Conversation

Akirathan commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Important Notes

Checklist

Uh oh!

Akirathan commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Akirathan commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Akirathan May 5, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 6, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach left a comment

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 6, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach May 6, 2025

Choose a reason for hiding this comment

Uh oh!

Akirathan commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

JaroslavTulach commented May 11, 2025

Uh oh!

Uh oh!

Akirathan commented Apr 30, 2025 •

edited

Loading

Akirathan commented Apr 30, 2025 •

edited

Loading

Akirathan commented May 5, 2025 •

edited

Loading

Akirathan commented May 6, 2025 •

edited

Loading