Memory usage grows exponentially of the esbuild process

[thomaschaaf]

When using serverless-offline and making changes to files the memory usage of the esbuild process seems to grow exponentially.

I start out with about 1 GB of ram usage and after 10 changes the esbuild process is using 8 GB of ram.

[olup]

I can confrim this. I am running into FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory continuously - and my colleagues on the project too.
It is probably linked to the alchemy between chokidar and esbuild.

[olup]

From what I can see this does the leak:

serverless-esbuild/src/index.ts

Line 160 in ad3e93e

this.bundle(true)

true option here is using incremental bundling. Maybe a lead.

[olup]

incremental build does keep things in memory to speed up subsequent bundling: https://esbuild.github.io/api/#incremental. Could it be a leak in esbuild itself? Should we call dispose to free up resources from time to time ?

Or we could leave the watching to esbuild itself : https://esbuild.github.io/api/#watch, and see if memory is better handled but I guess this was sub-optimal @floydspace ?

[thomaschaaf]

@olup I think it's just used incorrectly. We need to call rebuild() instead of initiating a new esbuild instance with build(). That way rebuilding would be even faster aswell.

[olup]

Yes true, otherwise the cache is a bit lost anyway. I'll flesh a pr tomorrow. Beyond this one subject, serverless offline, in watch mode, is terribly leaking so the rest of the process will end up eating the whole memory anyway. But we can improve what depends on us.

[thomaschaaf]

I rewrote it to use rebuild. Sadly this does not resolve the memory issue. #101 so I think it might actually be an esbuild issue.

But I am unable to verify it with:

async function example() {
  let result = await require('esbuild').build({
    entryPoints: ['src/index.ts'],
    bundle: true,
    outfile: 'out.js',
    incremental: true,
  });

  for (let i = 0; i < 150; i++) {
    console.log('a' + i);
    await result.rebuild();
  }

  console.log('done');
}

example();

[mattisdada]

I am reporting I've had similar issues, each build just linearly increases the amount of memory esbuild is using (about 2.5GB per file change, it pretty quickly gets up to 20GB), if I was to hazard a guess, I'm thinking it has something to do with node_modules and this quote from the esbuild docs:

Files are stored in memory and are not re-read from the file system if the file metadata hasn't changed since the last build. This optimization only applies to file system paths. It does not apply to virtual modules created by plugins.

I have a feeling something is causing all the metadata change from ESBuilds perspective and it might not garbage collect old metadata (I imagine it doesn't watch for that). But I haven't looked at any of the code for serverless-esbuild so I might be completely wrong...

incremental build does keep things in memory to speed up subsequent bundling: https://esbuild.github.io/api/#incremental. Could it be a leak in esbuild itself? Should we call dispose to free up resources from time to time ?

Or we could leave the watching to esbuild itself : https://esbuild.github.io/api/#watch, and see if memory is better handled but I guess this was sub-optimal @floydspace ?

I'd be curious what this responds back with when logging is set to max

I might experiment a little tonight with this

[anthony-zhou]

I've been experiencing the same problem of exponential growth in memory usage.

As far as I can tell, rebuild seems to have better memory usage results than calling build every time.

With the changes in my PR, memory usage seems to cap out around 1 GB (for my current codebase), whereas it used to increase exponentially until crashing.

Let me know if the PR #123 solves your problem @olup or @thomaschaaf.

[mattisdada]

That PR partially fixed the issue for me (it still uses a lot of memory, but not as much), but serverless-offline still consumes a lot of memory unless allowCache is enabled and eventually exhausts itself (and dies) - this may be an issue with our particular code base though. Not sure if others are experiencing this particular pain point.

The problem of enabling allowCache is that when ESBuild recompiles, the running code does not change to the newly compiled files, so to see changes it requires resetting the server or having a large memory leak.

I put in a very blunt workaround in the compiled serverless-esbuild files after the build that does seem to work wonderfully which is just:

Object.keys(require.cache).forEach(function(key) { delete require.cache[key] })

This patched version + using serverless-offline in allowCache mode, has allowed us to have best of both worlds. An "official" solution should only clear the cache of the compiled files, not everything like this...

Not sure if serverless-offline has an "official" way of bundlers to manage cache...

[olup]

Indeed, I put the link somewhere, but the memory leak in serverless offline is well known and affecting everyone, and the relation with the caching option is discussed around, but the accepted answer seems to be "that's how it is". So no official way to fix this.

A workaround in our codebase would be a very good idea, as the memory leak is a real pain, thanks a lot for sharing this ! Only we should probably put it behind some options and disabled by default, and document it.

[mattisdada]

Before a more official solution is in place, this is what we ended up using:

# serverless.yml
custom:
  serverless-offline:
    allowCache: true # Important: This will prevent serverless-offline from eating all of your memory
    # ...
  esbuild:
    plugins : ./esbuild/plugins.js
    # ....

// esbuild/plugins.js
let refreshServer = {
  name: "refresh-server",
  setup(build) {
    build.onEnd((result) => {
      if (build.initialOptions.incremental) {
        console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`);
        // Remove all items from the cache (this will force node to reload all of the built artifacts)
        Object.keys(require.cache).forEach(function (key) {
          const resolvedPath = require.resolve(key);
          if (resolvedPath.includes(build.initialOptions.outdir)) {
            delete require.cache[key];
          }
        });
      }
    });
  },
};
module.exports = [refreshServer];

By utilising the esbuild plugin system we just did the clearing there instead of in the serverless-esbuild side. We now have a watch that can run for a long time, that gets updated whenever we change code and our memory usage remains relatively static. Would strongly recommend for anyone running into any of these memory issues or having to restart serverless whenever code changes (because of attempts to resolve the memory issues).

Hope that helps someone else...

[olup]

Brilliant ! And it's much better than having inside the lib, indeed.
But it is such a central pb with serverless offline we should definitely link this in the doc!

[mattisdada]

Only small problem is the esbuild plugins don't get any serverless context, I'll continue playing around with this and see if I find any issues regarding that.

EDIT: Already ran into an issue with this if it's running during pack it'll cause some issue with the archiver lib (should hopefully be resolved by checking if incremental is enabled)

I made a few changes to the plugin to be much more selective instead of clearing out everything, it should only clear out entries in the esbuild output directory.