Suggestion: using elm-serialize

[MattiaVerticchio]

I want to make a suggestion, hoping it’s useful. 😁

I’ve read this response about the maximum supported model size of 166,6 MB (JSON encoded).

The package elm-serialize produces compressed JSON encoding.

There is a post on the Elm Discourse and it proved to be beneficial in elm-review too:

elm-serialize produces a significantly compressed JSON output (or binary, but which proved to be less efficient at this time). Instead of {"expression": {"type": "Operator", "operator" : "+" } } it will be something like [0,[0,0,1,"+"]].

Overall, this reduces the total cache size from 40MB to 14MB in the large project I mentioned, and because there is less data to read from (and to write to) the slow file system, the total run time was reduced by about 15%!

[Viir]

Thank you for sharing this idea. It looks like it would help to avoid that limit. As far as I see, that package uses a different interface than the JSON libraries. So would it require changing the compiler to emit different code?
An example of the Elm code produced by the current compiler is at 4834294#diff-f9ef3d37f5a52da0a2284ac48d4a91611741f93cb0269823ee2e8cbdb8916fcf

Mid-term, it looks like said limit would be lifted by a change in the upstream project (linked issue) or the switch of Elm Fullstack to the PineVM. PineVM has its own persistence infrastructure model anyway.

[peteygao]

Is this issue resolved in the short term via this commit and release 2023-11-15?

[Viir]

Is this issue resolved in the short term via this commit and release 2023-11-15?

I now ran a test with version 2023-11-15 to find out. I tested with database sizes up to 250 MB, and that worked. So that limit of 166,6 MB is resolved.

However, at the moment, the backend still relies on compiling Elm to JavaScript and running that in a JS engine. And JS engines introduce various bottlenecks. With version 2023-11-15, you will eventually run into limits related to the JavaScript approach with larger database sizes.

We made good progress this year with the new compiler and runtime for Elm/Pine and declarative programming languages in general.
Looks like we are on track to free Elm backends from JavaScript and JS engines next year.

[Viir]

I’ve read #18 (comment) about the maximum supported model size of 166,6 MB (JSON encoded).

An update on the scalability front: That bottleneck for database size is now resolved. I ran a test using version 0.4.3 and tested up to one Gigabyte Elm application state size without problems.
Here is the command used for the live test:

 pine  run-server  --process-store=./proc-store  --delete-previous-process  --deploy=https://github.com/pine-vm/pine/tree/422c9b632d41ad85649165062578142e0f52b6dd/implement/example-apps/test-database-scale

As discussed earlier, we switched from JavaScript to a better compilation target and recently removed the last dependencies on JavaScript.
Since the DB size limit resulted from interfacing with a JS engine, such a problem does not exist with the new engine.

The switch to an engine optimized for declarative programming will also increase response times and throughput.