Nostr sqldb

[tompro]

Description

Implements a basic driver for using Postgres as storage engine for nostr-database. Uses async Diesel as driver and Diesel migrations to setup the schema.

Checklist

• I followed the contribution guidelines
• I ran just precommit or just check before committing
• I updated the CHANGELOG (if applicable)

[yukibtc]

Thank you! I just had a quick look.

Since diesel support both PostgreSQL, SQLite and MySQL, maybe is better to rename this crate to something like nostr-sql (or maybe a better name) and add also the support for SQLite (in another PR). What do you think? Is the schema defined for PostgreSQL compatible also with SQLite, right?

I see that the SQLite implementation in diesel should work also in WASM, so this will allow to finally drop the nostr-indexeddb crate.

[tompro]

Yes sounds good, I will rename the module to nostr-sqldb and rework some of the Postgres specific code and naming. I neeed to run some tests whether all the stuff used is actually compatible with all backends (but it should as it is not too complex stuff).

There will probably be some breaking changes when I add multi db support later (probably need some feature flags and such) but this is alpha anyhow 😁 .

One other thing I was wondering is the filter fields. It would be nice to have a limit (max returned) and an order (asc, desc) for the timestamp field on it. Can I just add those fields to the database crate filter struct (non breaking with defaults)?

[yukibtc]

Yes sounds good, I will rename the module to nostr-sqldb and rework some of the Postgres specific code and naming. I neeed to run some tests whether all the stuff used is actually compatible with all backends (but it should as it is not too complex stuff).

Awesome, thank you!

One other thing I was wondering is the filter fields. It would be nice to have a limit (max returned) and an order (asc, desc) for the timestamp field on it. Can I just add those fields to the database crate filter struct (non breaking with defaults)?

There is already a limit field in the Filter. Regarding the order, in the previous versions of nostr-database there was the option to choose the query order, but I removed it because the queries in nostr are basically always DESC by timestamp. Is there some case where is useful to query ASC?

[tompro]

Ah yes, sorry I meant offset not limit (but it is not that important as timestamp based chunks work well enough). The ASC order on the other hand would be great for querying groups or threads from start to finish. I think the DESC order is very useful for the 'Twitter' like use case but for conversational sorts of messages ASC would work a lot better.

[yukibtc]

The ASC order on the other hand would be great for querying groups or threads from start to finish. I think the DESC order is very useful for the 'Twitter' like use case but for conversational sorts of messages ASC would work a lot better.

Ok, thanks. I'll evaluate to revert the commit where I remove the Order arg from the query method.

For now I would say to extend the nostr-sqldb as you prefer, adding ways to perform more advanced queries, and we'll see if there is a good way to generalize it for the NostrDatabase traits. And, if there isn't a good way to generalize it, exists always the possibility of downcasting the Arc<dyn NostrDatabase> to the specific nostr-sqldb database struct when needed, which could be useful also for other databases like nostrdb which exposes custom APIs.

[tompro]

I added db specific migrations and generated schema (mappers) but no NostrDb traits impl for now. The PG implementation is now in its own module. After testing around with the feature flags I came to the conclusion that all queries need to be built completely generic (db backend seeps into connection and model mapping so the I an O of it 😁). My idea is to do a generic impl. for both Mysql and Sqlite and then migrate the PG version to it as well. NostrPostgres::new(...) now also runs any pending migrations.

[yukibtc]

I would store the 32-byte array here, instead of the hex. The same for the pubkey and signature (64-byte array), also in the other schemas. What do your think?

[tompro]

I'll have to check whether there is support for binary indexes in all dbs and if there are any performance penalties. First look seem to be doable. Debugging the data (with other sql clients) will get a bit harder though.

[yukibtc]

Regarding the debugging in SQL clients: I know some clients display the bytes as hex, but not sure if all. I've tried DBeaver and JetBrains Database tool, and both display the event ID and public key BLOBs as hex.

[tompro]

Yes at least PG is storing small binaries as hex values anyhow. I have dropped the signature column and set the column types to be binary. Testing with PG seems to have no issues for the other backends I'll have to check when I add the impls.

[yukibtc]

Thanks for the changes. I tested it locally and for now I prefer to not merge it in master branch yet. I'm going to move to a sqldb branch, so we can work on it and merge to master when all backends are supported.

[yukibtc]

Squashed and moved to sqldb branch, so for new changes open PRs to that branch.
I opened the new PR for the branch here: #855