Bug: Snowpack Javascript heap out of memory during "installing dependencies..." #1548

Kleptine · 2020-11-08T14:58:41Z

Kleptine
Nov 8, 2020

Snowpack takes 80 seconds to finish "installing dependencies", every time I touch the package.json file. Since this morning, it no longer finishes, crashing and running out of memory:

My project uses the yFiles library, which is distributed as a collection of very large, minified and obfuscated javascript files. I imagine these files are the problem, but these types of dependencies are pretty common when you're working with proprietary/licensed libraries.

Priority: Snowpack is Unuseable

Kleptine · 2020-11-08T15:03:44Z

Kleptine
Nov 8, 2020
Author

yFiles is already distributes as an ES6 Module. So what is Snowpack doing during 'installing dependencies.." ?

3 replies

mikaelbalin Nov 9, 2020

It converts modules to ESM https://www.snowpack.dev/#import-npm-packages

Kleptine Nov 9, 2020
Author

Right, but in this case, it's already an ESM module! (I think). So I believe it should just be doing nothing. 🤔

FredKSchott Dec 7, 2020
Maintainer

It shouldn't be doing anything too complex, but we do still bundle multi-file ESM into a single package-name.js file for importing in the browser.

mikaelbalin · 2020-11-09T20:54:44Z

mikaelbalin
Nov 9, 2020

The same for me. Is there any option to write web_modules to disc, why it should be in memory?

1 reply

Kleptine Nov 9, 2020
Author

I have found that web_modules is in node_modules/.cache/snowpack/. Those files seem to regenerate when I start the dev server.

nurseiit · 2020-11-17T11:52:36Z

nurseiit
Nov 17, 2020

The same happens with a fairly small TypeScript/React codebase.
Snowpack version ^2.17.1.

More context:
npx snowpack install finishes alright, snowpack dev is what's failing.

9 replies

Kleptine Nov 17, 2020
Author

For powershell: $Env:NODE_OPTIONS="--max-old-space-size=4096" although I had to use 8192 in my case.

FredKSchott Dec 7, 2020
Maintainer

Thanks everyone for jumping in here, would love to get this fixed. We've heard a few reports of this, but no one has been able to generate a useful heapdump to investigate these memory issues with. If anyone is able to run through these steps on their machine to generate one, that would be amazingly useful: https://medium.com/@wavded/how-to-heap-snapshots-aac9284d5329

rizrmd Feb 8, 2021

I'm doing export NODE_OPTIONS="--max-old-space-size=4096" && yarn dev which is not quite ideal tbh.

this does not prevent mine from crashing...

jonrimmer Feb 17, 2021

@FredKSchott I tried following the instructions in the page you linked and they did not work. The heapdump produced is empty. This is a known caveat of Heapdump. You cannot produce heapdumps of massive, multi-gigabyte heaps like Snowpack produces.

Do you have any other suggestions about how to debug this?

sepehr500 Apr 23, 2021

Setting the max size worked for me. If you want to reproduce this, try installing @arcgis/core and importing import MapView from "@arcgis/core/views/MapView"

rizrmd · 2021-02-08T01:45:38Z

rizrmd
Feb 8, 2021

Attached at the bottom are my heap snapshot and error log:

<--- Last few GCs --->

[16788:00000299DC148080] 1311485 ms: Scavenge 4045.5 (4088.9) -> 4045.2 (4089.4) MB, 3.4 / 0.0 ms (average mu = 0.908, current mu = 0.472) allocation failure
[16788:00000299DC148080] 1311582 ms: Scavenge 4049.8 (4090.1) -> 4047.9 (4091.6) MB, 2.7 / 0.0 ms (average mu = 0.908, current mu = 0.472) allocation failure
[16788:00000299DC148080] 1311665 ms: Scavenge 4051.5 (4091.9) -> 4049.9 (4093.4) MB, 3.8 / 0.0 ms (average mu = 0.908, current mu = 0.472) allocation failure

<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
1: 00007FF6D294136F napi_wrap+109151
2: 00007FF6D28E6716 v8::internal::OrderedHashTablev8::internal::OrderedHashSet,1::NumberOfElementsOffset+33350
3: 00007FF6D28E74E6 v8::internal::OrderedHashTablev8::internal::OrderedHashSet,1::NumberOfElementsOffset+36886
4: 00007FF6D31B58EE v8::Isolate::ReportExternalAllocationLimitReached+94
5: 00007FF6D319A73D v8::SharedArrayBuffer::Externalize+781
6: 00007FF6D3044B7C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
7: 00007FF6D302F71B v8::internal::NativeContextInferrer::Infer+59339
8: 00007FF6D3014B4F v8::internal::MarkingWorklists::SwitchToContextSlow+57503
9: 00007FF6D30288AB v8::internal::NativeContextInferrer::Infer+31067
10: 00007FF6D301F93D v8::internal::MarkCompactCollector::EnsureSweepingCompleted+6269
11: 00007FF6D3027AFE v8::internal::NativeContextInferrer::Infer+27566
12: 00007FF6D302BA7B v8::internal::NativeContextInferrer::Infer+43819
13: 00007FF6D30352B2 v8::internal::ItemParallelJob::Task::RunInternal+18
14: 00007FF6D3035233 v8::internal::ItemParallelJob::Run+643
15: 00007FF6D30089B3 v8::internal::MarkingWorklists::SwitchToContextSlow+7939
16: 00007FF6D301FDFC v8::internal::MarkCompactCollector::EnsureSweepingCompleted+7484
17: 00007FF6D301E634 v8::internal::MarkCompactCollector::EnsureSweepingCompleted+1396
18: 00007FF6D301C148 v8::internal::MarkingWorklists::SwitchToContextSlow+87704
19: 00007FF6D304A921 v8::internal::Heap::LeftTrimFixedArray+929
20: 00007FF6D304CA05 v8::internal::Heap::PageFlagsAreConsistent+789
21: 00007FF6D3041C21 v8::internal::Heap::CollectGarbage+2033
22: 00007FF6D303FE25 v8::internal::Heap::AllocateExternalBackingStore+1365
23: 00007FF6D3060347 v8::internal::Factory::NewFillerObject+183
24: 00007FF6D2D8DBA1 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1409
25: 00007FF6D323E72D v8::internal::SetupIsolateDelegate::SetupHeap+463949
26: 00007FF6D328F2CF v8::internal::SetupIsolateDelegate::SetupHeap+794607
27: 00007FF6D322F522 v8::internal::SetupIsolateDelegate::SetupHeap+401986

Heap-20210208T084354.zip

1 reply

rizrmd Feb 8, 2021

This happens every time when I develop on my windows machine...

Kleptine · 2021-02-08T09:05:32Z

Kleptine
Feb 8, 2021
Author

Try using >8192.

…

On Mon, Feb 8, 2021 at 1:43 AM Rizky ***@***.***> wrote: I'm doing export NODE_OPTIONS="--max-old-space-size=4096" && yarn dev which is not quite ideal tbh. this does not prevent mine from crashing... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1548 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIHMCHP5VFQHMB3JT7AXN3S56P4LANCNFSM4TOLXD4A> .

2 replies

rizrmd Feb 8, 2021

So.... the only solution is to just increase memory limit ?

rizrmd Feb 8, 2021

No, it's stilll happen... but it holds longer and then crash.

artemis-prime · 2021-02-14T15:37:54Z

artemis-prime
Feb 14, 2021

Increasing the limit is not a solution. I get a spinning cursor in the browser window and then the crash (after a couple of minutes). I'm dead in the water.

0 replies

JoeMethven · 2021-09-10T13:23:43Z

JoeMethven
Sep 10, 2021

Has anyone found a solution to this? Mine crashes even if I set to 8192 limit...

0 replies

AkashKumarVerma · 2021-12-09T04:31:02Z

AkashKumarVerma
Dec 9, 2021

Anybody ... Getting same problem
WSL2
node version 16.13.1

Here is the error

`[10:56:15] [snowpack] Ready!
[10:56:15] [snowpack] Server started in 34ms.
[10:56:15] [snowpack] Local: http://localhost:8080
[10:56:15] [snowpack] Network: http://172.21.197.49:8080
⠇ watching for file changes...
<--- Last few GCs --->

[5186:0x4d813a0] 80163 ms: Scavenge (reduce) 941.0 (969.6) -> 941.0 (969.6) MB, 4.7 / 0.0 ms (average mu = 0.130, current mu = 0.070) allocation failure
[5186:0x4d813a0] 80169 ms: Scavenge (reduce) 941.9 (970.6) -> 941.9 (970.6) MB, 3.6 / 0.0 ms (average mu = 0.130, current mu = 0.070) allocation failure
[5186:0x4d813a0] 80173 ms: Scavenge (reduce) 943.3 (972.0) -> 943.3 (972.0) MB, 3.4 / 0.0 ms (average mu = 0.130, current mu = 0.070) allocation failure

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xb00d90 node::Abort() [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
2: 0xa1823b node::FatalError(char const*, char const*) [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
3: 0xcedbce v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
4: 0xcedf47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
5: 0xea6105 [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
6: 0xea6be6 [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
7: 0xeb4b1e [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
8: 0xeb5560 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
9: 0xeb84de v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
10: 0xe7990a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]
11: 0x11f2f06 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/deiknymi/.nvm/versions/node/v16.13.1/bin/node]`

0 replies

Bug: Snowpack Javascript heap out of memory during "installing dependencies..." #1548

Uh oh!

Uh oh!

Replies: 8 comments · 16 replies

Uh oh!

Uh oh!

Kleptine Nov 8, 2020 Author

Uh oh!

Uh oh!

Kleptine Nov 9, 2020 Author

Uh oh!

FredKSchott Dec 7, 2020 Maintainer

Uh oh!

Uh oh!

Kleptine Nov 9, 2020 Author

Uh oh!

Uh oh!

Kleptine Nov 17, 2020 Author

Uh oh!

FredKSchott Dec 7, 2020 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kleptine Feb 8, 2021 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 8 comments 16 replies

Kleptine
Nov 8, 2020
Author

Kleptine Nov 9, 2020
Author

FredKSchott Dec 7, 2020
Maintainer

Kleptine Nov 9, 2020
Author

Kleptine Nov 17, 2020
Author

FredKSchott Dec 7, 2020
Maintainer

Kleptine
Feb 8, 2021
Author