Separate "client" and "server" ?

[cyrilchapon]

Hello !

I just spent several days deploying a lambda with duckdb-node-neo inside; and was surprised to see that this was not less-complicated to do so if I need like a "full duckdb instance" (I mean, the "core" of DuckDB) in my lambda than "just the connector on which I can give a URL of a DuckDB instance".

I don't understand everything about DuckDB still; but coming from several "DB like technologies", I was expecting to be able to connect to a MotherDucks (or another provider) instance with a URL and just throw a "duckdb connector" in my lambda or anywhere else; and was expecting the majority of the DuckDB code to be actually inside MotherDucks itself, with a minimal implementation on the client side.

I feel like it's a bit blurry and again I might not really be understanding everything; but is there any logical opportunity for a separation of concerns between what I call a "client" and a "server" ?

[jraymakers]

Although your assumptions about the separation between "client" and "server" are natural and reasonable, as that is how other databases often work, it is not how DuckDB or MotherDuck work.

DuckDB is an embedded database, which means the database runs directly in the process in which it is used. This is in contrast to many other database systems, where the database runs in a server process that is separate from the client.

MotherDuck embraces this architecture. Although MotherDuck does run an instance of DuckDB on a server (in the cloud), a MotherDuck client also runs an instance of DuckDB. These two instances communicate, and both are capable of running queries, or parts of queries. This is what is known as dual execution; it enables processing to occur closest the data.

Getting the most out of this atypical architecture does require some changes in how you build applications, but it can be very powerful.