official aws lambda layer for @duckdb/node-api

[cookiejest]

Hey, is there an official aws lambda layer for @duckdb/node-api or we must include as part of our own deployment pipeline?

Currently getting this error on a clean lambda function. I guess most people would also get this:

   "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '@duckdb/node-bindings-linux-x64/duckdb.node'\nRequire stack:\n- /var/task/index.mjs",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module '@duckdb/node-bindings-linux-x64/duckdb.node'",

Many thanks!

[jraymakers]

There is not. The lambda layer for the old Node client isn't "official", either; it's provided by a member of the community.

I'm not very familiar with lambda layers so I don't know what's involved. I don't understand enough about the lambda environment to know why you got the error above, but my guess is there's something different about the Node version or configuration.

I'd welcome any concrete suggestions or PRs to make this work. I have a fair amount of other items on my list right now, so I'm not likely to have time to look into this anytime soon.

[cookiejest]

figured it out actually. It was due to lambda function running on a different chip to my local machine and the .node binding dependency not being installed when the function was deployed on aws.

I wouldn't want to use a random lambda layer as abit of a security risk but these are the steps i did to get it working (for others wanting to fix the same issue):

1. Confirm what lambda run time your using (if nodejs its probably linux64)
2. Clone your node_modules/@duckdb directory locally and delete the large files you dont need as it needs be below 250mb uncompressed and under 50mb compressed to be uploadable as a lambda layer. (I deleted duckdb-wasm and the binding that was for my local arm machine).
3. Get the correct binding for the environment the lambda function is running on (can be forced like this):
   npm i @duckdb/node-bindings-linux-x64 --force
   Here was mine:

5. zip it up and upload as a lambda layer and assign it to the lambda function

Now duckdb will be executable inside a lambda function without using a custom docker image.

[jraymakers]

Nice!

I can confirm that those three packages (@duckdb/node-api, @duckdb/node-bindings, and @duckdb/node-bindings-linux-x64) should be the minimum needed (for Linux x64).

[jraymakers]

Probably worth documenting this and adding it to the README. I'll keep this issue open to track that.

[cookiejest]

good idea! I think this would allow alot more people who are just using front end frameworks like aws amplify to use duckdb in their lambda functions :)

[hannesj]

When bundling a lambda function with SST v2, the installation of the native binary can be can be configured as follows:

  new Function(stack, "InvoiceFunction", {
    handler: "functions/invoice.handler",
    nodejs: {
      esbuild: {
        loader: { ".node": "file" },
      },
      install: ["@duckdb/node-bindings-linux-arm64"],
    },
  });

[cookiejest]

That's much better !🤗 (if your using sst)

[subos2008]

What's the use case for DuckDB in lambdas? Given they throw their memory and filesystem away after each call. You could mount a filesystem into the lambda but that wouldn't be safe unless invocation concurrency was hard coded to 1.

Asking because I'm porting from DuckDB to DynamoDB for AWS deployment into lambdas myself as I couldn't see a way to use DuckDB in lambdas

[cyrilchapon]

It depends on if we are talking about DuckDB "client" or "server".
Use cases for what I call server (in memory, all in one) are limited; but use cases for what I call client-only can be massive. I'm landing here after trying and deploy a lambda with DuckDB nodejs connector embedded inside a lambda, which is actually targetting a Motherducks instance.

[tobilg]

That's not entirely correct, Lambda functions have a context that might get recycled. If the functions stays "warm", i.e. is used, it will keep it's memory persisted between invocations by the underlying engine. AWS doesn't publish the exact conditions when contexts get recycled...

The main use case I use DuckDB in Lambda functions (with https://github.com/tobilg/duckdb-nodejs-layer) is for adhoc queries towards data in S3.

[tobilg]

Furthermore, if you'd want to persist the database file between invocations (and even context recylcles), you could use EFS to mount a remote filesystem to a local path

[cookiejest]

Hi, I am using it for scaled querying on s3 hosted parquet files. Actually the dB are created in read only mode and created as part of the single query with a unique dB name. Even if the memory was shared between lambda calls. The dB are completely different and units to that evocation. This is viable because the data is small and compressed already

[cookiejest]

The primary use is if your data is already in S3 or s3 tables and you have a complex analytical query your running on every request you don't want a separate database for (or your trying to reduce athena use), you can use the /tmp folder to go out of memory if you really need too.

Using dynamodb for analytical queries would be a world of pain. You would be better with aurora but then there is more to manage and has its own problems.

[cyrilchapon]

Hello;

After struggling for almost 2 days to deploy that on a AWS lambda + serverless context; I feel like it's worth it to share experience, potentially being the base of a future layer.

A bit of context :

• Serverless v4
• Node 20 runtime
• yarn
• Core serverless build.esbuild with actual ESM setup (type: "module") (no plugin for building)
• individually: true

---

Fails

I made an initial attempt first, having in mind "not deploying a layer at all"; and faced several troubles, with one of them actually being unresolvable.

1. Don't touch anything and let package + esbuild do their thing
   The trouble with that is, depending on your dev environment, yarn will be installing the relevant version of @duckdb/node-bindings-[plaform-specific-one] (darwin + arm64 in my case); whereas the default lambda one have to be linux + x64.

Of course, we could go crazy and configure the lambda runtime to match the OS one or build inside a VM or something; but that feels hacky as hell and is not scalable by any means.

2. Try and install locally and then force deploy, a different architecture version than my local one
   Just go for yarn add @duckdb/node-bindings-[plaform-specific-one] with a different one than on your machine, will all possible arguments I found on the web (--arch, --platform, npm_config_arch, npm_config_platform, --target, all of this inline or inside .npmrc, yarnrc.yml) never worked. Yarn just absolutely refuses to install this locally.

I discovered that npm can do it, with --force, but then you have to switch your whole environment to npm, and execute everything with npm from the build to the deploy going through the packaging. Not OK for me

3. Auto-package in a layer, with the help of a plugin
   Good idea, but it actually uses the local one to package, so back to problem 1; that doesn't help.

---

Success

After hours of researches and testing, I succeeded on deploying everything with the bindings inside a layer, with the following setup :

serverless.yml

frameworkVersion: '4'

provider:
  name: aws
  runtime: nodejs20.x
  # ...
  environment:
    # Needed despite what AWS documentation says
    NODE_PATH: './:/opt/nodejs/node_modules'

layers:
  duckdb:
    # The local path of the layer folder to build
    path: ./duckdb-layer
    description: DuckDB bindings for Node.js

package:
  individually: true
  patterns:
    # Those ones are DuckDB bindings for specific functions
    # they are extracted in ./duckdb-layer
    - '!node_modules/@duckdb/node-bindings'
    - '!node_modules/@duckdb/node-bindings-darwin-arm64'
    - '!node_modules/@duckdb/node-bindings-darwin-x64'
    - '!node_modules/@duckdb/node-bindings-linux-arm64'
    - '!node_modules/@duckdb/node-bindings-linux-x64'
    - '!node_modules/@duckdb/node-bindings-win32-x64'

build:
  esbuild:
    # See below
    configFile: ./esbuild.config.js

functions:
  sync-subpage-user-aggregate:
    # ... Function configuration
    layers:
      # Reference the layer to use it
      - !Ref DuckdbLambdaLayer

esbuild.config.js

/**
 * @import {BuildOptions} from "esbuild"
 */

/** @type{() => BuildOptions} */
export default () => ({
  bundle: true,
  minify: true,
  sourcemap: false,
  keepNames: true,
  format: 'esm',
  platform: 'node',
  banner: {
    // This hack will fix actual lambda ESM environment
    // versus esbuild require-wrapping logic (basically make `require` work)
    js: "import { createRequire as topLevelCreateRequire } from 'module';\n const require = topLevelCreateRequire(import.meta.url);",
  },
  // Mark bindings as external (not embedded inside the function code)
  external: [
    '@duckdb/node-bindings',
    '@duckdb/node-bindings-darwin-arm64',
    '@duckdb/node-bindings-darwin-x64',
    '@duckdb/node-bindings-linux-arm64',
    '@duckdb/node-bindings-linux-x64',
    '@duckdb/node-bindings-win32-x64',
  ],
  // And actually exclude them totally from the build
  exclude: [
    '@duckdb/node-bindings',
    '@duckdb/node-bindings-darwin-arm64',
    '@duckdb/node-bindings-darwin-x64',
    '@duckdb/node-bindings-linux-arm64',
    '@duckdb/node-bindings-linux-x64',
    '@duckdb/node-bindings-win32-x64',
  ]
})

And now here comes the funny part; creating the layer.

Create base layer folder

We're going to create the layer inside ./duckdb-layer as specified in the serverless.yml

I actually used npm for this, with that exact command :

npm install --prefix ./duckdb-layer @duckdb/node-bindings @duckdb/node-bindings-linux-x64 --force

With --prefix, you're installing that into a new subdirectory, and not messing with your upper yarn (or whatever) configuration. With --force you're shuting npm up from saying "hey you're installing an incompatible architecture package".

Inside ./duckdb-layer it automatically installed

• @duckdb/node-bindings (because specified)
• @duckdb/node-bindings-linux-x64 (because specified)
• @duckdb/node-bindings-darwin-x64 (because needed for my dev environment)
  ...and initialized a bunch of package.json, package-lock.json etc.

[Optional] Remove dev-platform specific bindings

From there, I had to clean that from having unwanted bindings for darwin-x64 because as stated previously it installed it automatically, so I

• deleted ./duckdb-layer/node_modules/node-bindings-darwin-x64
• and removed relevant parts in package-lock.json

Fix file architecture

Despite what AWS configuration docs says, if you deploy like this, your lambda won't actually find the module inside ./duckdb-layer/node_module; because of the node path apparently being broken.

You would end up with

ImportModuleError: [...] Cannot find module @duckdb/node-bindings

The exact file architecture that works for me is :

• duckdb-layer
  • nodejs
    • node_modules
    • package.json
    • etc...

So basically cut-paste ./duckdb-layer/* inside ./duckdb-layer/nodejs/

Fix NODE_PATH

Despite what AWS configuration docs says, if you deploy like so your lambda still won't find the module.

You would still end up with

ImportModuleError: [...] Cannot find module @duckdb/node-bindings

For this to work, you have to adjust the NODE_PATH to target the nested nodejs folder inside layers.
Good news, you already did, in the previous serverless.yml in environment section.

[cyrilchapon]

For what it's worth for anyone landing here; I created a request inside duckdb-nodejs-layer in the hope to make happen a new public layer with the official new duckdb neo bindings. If anyone feel to give that a starting point or help on that, that would be awesome.

[tobilg]

I commented on the other issue as well. Is it verified that the default DuckDB Linux binaries work in the AWS Lambda runtime environments? That would IMO mean that either DuckDB "downgraded" their used GLIBC version, or the contained (Node >=20) runtimes of Amazon Linux 2023 have upgraded their GLIBC version... I pretty sure I tested with Node 20 before and the default binaries didn't work. It's been a few months though.