feat(sqlite): SqliteEventCacheStore has 1 write connection

[Hywan]

Until now, SqliteEventCacheStore manages a pool of connections. A connection is fetched from this pool and operations are executed on it, regardless whether these are read operations or write operations.

We are seeing more and more database is busy errors. We believe this is because too many write operations are executed concurrently.

The solution to solve this is to use multiple connections for read operations, and a single connection for write operations. That way, concurrent writings are no longer a thing, and we hope it will reduce the number of database is busy errors to zero. That's our guess.

This patch does that. When the pool of connections is created, a connection is elected as the write_connection. To get a connection for read operations, one has to use the new SqliteEventCacheStore::read method (it replaces the acquire method). To get a connection for write operations, one has to use the new SQliteEventCacheStore::write method. It returns a OwnedMutexGuard from an async Mutex. All callers that want to do write operations on this store have to wait their turn, this Mutex is fair, and the first to wait on the lock is the first that will take the lock (FIFO). It guarantees the execution ordering the code expects.

The rest of the patch updates all spots where acquire was used and replaces them by read() or write(). A particular care was made to see if other places are using SqliteEventCacheStore::pool directly. No place remains except in read() and write().

Finally, other commits are adding more logs and timer logs.

---

• Address https://github.com/element-hq/customer-success/issues/591

[Hywan]

Bonus: I'm thinking about it right now, but we may want to add logs around write() to know how much it took to acquire the write connection. It could be a very useful information to debug slowness if any. I'll do that later. Please don't merge this PR immediately.

[Hywan]

This change ensures with_immediate_transaction takes a write connection.

[codecov]

Codecov Report

Attention: Patch coverage is 78.20513% with 17 lines in your changes missing coverage. Please review.

Project coverage is 88.80%. Comparing base (72d1332) to head (fd73cb3).
Report is 37 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
crates/matrix-sdk-sqlite/src/event_cache_store.rs	75.71%	3 Missing and 14 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5382      +/-   ##
==========================================
- Coverage   88.81%   88.80%   -0.01%     
==========================================
  Files         334      334              
  Lines       91143    91153      +10     
  Branches    91143    91153      +10     
==========================================
  Hits        80948    80948              
- Misses       6369     6385      +16     
+ Partials     3826     3820       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Hywan]

Quoting myself (because why not):

Bonus: I'm thinking about it right now, but we may want to add logs around write() to know how much it took to acquire the write connection. It could be a very useful information to debug slowness if any. I'll do that later. Please don't merge this PR immediately.

Done. I've added #[instrument] + timer! logs, almost everywhere. It will help spot slowness more easily.

[poljar]

Looks good, left a question.