JavaScript out of memory Error

[rrolonNelson]

Hello! Has anyone encountered a memory issue on 11ty??

Our team has been working on converting a hugo quire catalog to the 11ty Quire. We had several customizations on the Hugo version but have stripped most of those out while keeping a limited amount of customizations that could be translated to the 11ty version. The content team has been adding content and updating shortcodes and syntax since we made sure some of the customizations were working. They encountered some errors so I was going through and fixing the errors that were preventing the project to preview (most due to improper closing of HTML tags in md files).

Once I completed those I got an error similar to the one below saying that JavaScript had run out of memory. I increased my machine's node memory limit to 8000mb, which did not help (I have not tried adding node --max-old-space-size=8000 to any scripts yet). I did some research and found this github issue that said this error could be due to improper templates or an infinite loop with the templates being called inside each other. That did not seem to be the issue.
I found these other issues (large data set) (large data set & 11ty version) (large data set & pagination) which point to potentially the 11ty version needing to change (the version is 2.0 so I don't think that is the issue) or there being too many files or there being an issue with pagination (not sure if that is what I am dealing with either).

I think the issue is with the number of markdown files because when I realized this happened I went back in the commits to a version that was able to preview and added the fixed content files back in to see if there was one file causing a problem but it seems to happen when there is a certain amount of files added. AND it kind of depends on where the files are located. We have close to 400 md files and I started to see the issue pop up after successfully adding and being able to preview about 140 md files. It also starts to get a memory issue when adding files inside of folders that are inside folders in the content folder. ie content/British/Georgian-Era/44-54.md but when I add them to the root it will not produce the memory error up to a certain point. It doesn't seem to matter which files I add, it just doesn't seem to like if we add an excess of files.

I have tried this on Windows 10 and Mac M1 Monterey. With the same results.

The error happens after the figures processing is complete"
[quire] INFO Figures Processing complete

<--- Last few GCs --->

[32473:0x150040000] 103604 ms: Mark-sweep 4015.8 (4136.2) -> 4003.4 (4133.9) MB, 2973.3 / 0.0 ms (average mu = 0.093, current mu = 0.024) allocation failure; scavenge might not succeed
[32473:0x150040000] 106448 ms: Mark-sweep 4019.6 (4134.2) -> 4006.5 (4137.2) MB, 2793.5 / 0.0 ms (average mu = 0.057, current mu = 0.018) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x1006c54fc node::Abort() [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
2: 0x1006c56ec node::ModifyCodeGenerationFromStrings(v8::Localv8::Context, v8::Localv8::Value, bool) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
3: 0x10081b0fc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
4: 0x1009c6088 v8::internal::EmbedderStackStateScope::EmbedderStackStateScope(v8::internal::Heap*, v8::internal::EmbedderStackStateScope::Origin, cppgc::EmbedderStackState) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
5: 0x1009c9c90 v8::internal::Heap::CollectSharedGarbage(v8::internal::GarbageCollectionReason) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
6: 0x1009c6c84 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
7: 0x1009c40dc v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
8: 0x1009b8e14 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
9: 0x1009b9644 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
10: 0x10099f478 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
11: 0x100d2f2bc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
12: 0x10107904c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
13: 0x10104f300 Builtins_MapConstructor [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
14: 0x101001914 Builtins_JSBuiltinsConstructStub [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
15: 0x1066a6704
16: 0x106c64e1c
17: 0x1062e3060
18: 0x1062df404
19: 0x106c74fc4
20: 0x1062d2068
21: 0x10621f7d8
22: 0x106aa0ae8
23: 0x106aa0c50
24: 0x106aa0e4c
25: 0x106ab7458
26: 0x106aa08e4
27: 0x101001700 construct_stub_create_deopt_addr [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
28: 0x106c24070
29: 0x106c3bfb0
30: 0x106309968
31: 0x106bf2398
32: 0x106bf2600
33: 0x106309968
34: 0x10649b460
35: 0x101035ef4 Builtins_AsyncFunctionAwaitResolveClosure [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
36: 0x1010c4738 Builtins_PromiseFulfillReactionJob [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
37: 0x101027c4c Builtins_RunMicrotasks [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
38: 0x1010023a4 Builtins_JSRunMicrotasksEntry [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
39: 0x10094782c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
40: 0x100947d1c v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
41: 0x100947ef8 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandlev8::internal::Object) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
42: 0x10096e644 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
43: 0x10096eedc v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
44: 0x100614c50 node::InternalCallbackScope::Close() [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
45: 0x1006147c4 node::InternalCallbackScope::~InternalCallbackScope() [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
46: 0x1006c8cd4 node::fs::FileHandle::CloseReq::Resolve() [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
47: 0x1006df974 node::fs::FileHandle::ClosePromise()::$_0::__invoke(uv_fs_s*) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
48: 0x1006c1330 node::MakeLibuvRequestCallback<uv_fs_s, void ()(uv_fs_s)>::Wrapper(uv_fs_s*) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
49: 0x100fe1058 uv__work_done [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
50: 0x100fe4800 uv__async_io [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
51: 0x100ff6fa4 uv__io_poll [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
52: 0x100fe4cd0 uv_run [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
53: 0x1006156f4 node::SpinEventLoop(node::Environment*) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
54: 0x1007028ec node::NodeMainInstance::Run() [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
55: 0x100692028 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
56: 0x1006922b4 node::Start(int, char**) [/Users/rrolonNAMA/.nvm/versions/node/v18.16.0/bin/node]
57: 0x105c9108c

[geealbers]

@rrolonNelson I have run into this error myself in a project with a lot of large images and a lot of shortcode usage. To test further, I created a dummy project with 500 markdown files at the root level of the content directory, and got the same error. So yes, just the number of Markdown files can be an issue. I wasn't able to get it to run even when increasing the --max-old-space-size to 12GB. However, I just got a new Mac with 64GB of memory and was able to get past the memory issue when I increased the --max-old-space-size to 48GB. (I didn't try anything lower.) So, it's at least possible. Do you have more memory than the 8GB you said you tried, that you could try again?

We're also working now on getting this documented in the Troubleshooting section of the docs, and will post it as an issue so we can discuss internally about what can be done to optimize Quire's memory usage.

[rrolonNelson]

I was able to up my Windows machines ram to 64GB, setting the machines environment variable NODE_OPTIONS to --max-old-space-size=4800 and adding NODE_OPTIONS=--max-old-space-size=48000 inside of the serve and dev script in package.json and running npm run serve or npm run dev. (if I run quire preview I continue to get the "JavaScript heap out of memory" error"). It takes approx. 30GB of memory to build a preview.

[Erin-Cecele]

Just a note that this has been documented as issue #748