increase threadpool size to avoid thread starvation (#8686)

We’ve seen some symptoms of thread starvation (health check timeouts
despite systems looking normal).

It is plausible that 300 threads may get stuck in synchronous waiting,
e.g. for high-latency remote GCS endpoints.

Increasing the thread pool size helps mitigate this problem to some
extent. A better solution would presumably be separate thread pools
#8685

### URL of deployed dev instance (used for testing):
- https://morethreads.webknossos.xyz

------
- [x] Updated
[changelog](../blob/master/CHANGELOG.unreleased.md#unreleased)
- [x] Needs datastore update after deployment

Author: fm3

MIGRATIONS.unreleased.md

@@ -8,5 +8,7 @@ User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 ## Unreleased
 [Commits](https://github.com/scalableminds/webknossos/compare/25.06.1...HEAD)
 
+- The default thread pool size was increased from 5 to 10 times the number of available CPUs (capped at 1000). Note that wk may need slightly more memory because of this. [#8686](https://github.com/scalableminds/webknossos/pull/8686)
+
 ### Postgres Evolutions:
 - [134-dataset-layer-attachments.sql](conf/evolutions/134-dataset-layer-attachments.sql)

conf/application.conf

@@ -66,12 +66,12 @@ play {
 pekko.actor.default-dispatcher {
   # We use a compromise for our thread pool configuration
   # Parts of our api are async, so they should not need many threads,
-  # but some parts are also blocking (file io, gcs, s3 access), causing new requests
+  # but some parts are also blocking (some file io, gcs access), causing new requests
   # to wait despite idle cpu, if there are too few threads
   fork-join-executor {
-    parallelism-factor = 5.0 # Thread count = ceil(available processors * factor)
+    parallelism-factor = 10.0 # Thread count = ceil(available processors * factor)
     parallelism-min = 8 # Min number of threads to cap factor-based parallelism number to
-    parallelism-max = 300 # Max number of threads to cap factor-based parallelism number to
+    parallelism-max = 1000 # Max number of threads to cap factor-based parallelism number to
   }
 }
 

webknossos-datastore/conf/standalone-datastore.conf

@@ -38,12 +38,12 @@ play {
 pekko.actor.default-dispatcher {
   # We use a compromise for our thread pool configuration
   # Parts of our api are async, so they should not need many threads,
-  # but some parts are also blocking (file io, gcs, s3 access), causing new requests
+  # but some parts are also blocking (some file io, gcs access), causing new requests
   # to wait despite idle cpu, if there are too few threads
   fork-join-executor {
-    parallelism-factor = 5.0 # Thread count = ceil(available processors * factor)
+    parallelism-factor = 10.0 # Thread count = ceil(available processors * factor)
     parallelism-min = 8 # Min number of threads to cap factor-based parallelism number to
-    parallelism-max = 300 # Max number of threads to cap factor-based parallelism number to
+    parallelism-max = 1000 # Max number of threads to cap factor-based parallelism number to
   }
 }
 

webknossos-tracingstore/conf/standalone-tracingstore.conf

@@ -38,12 +38,12 @@ play {
 pekko.actor.default-dispatcher {
   # We use a compromise for our thread pool configuration
   # Parts of our api are async, so they should not need many threads,
-  # but some parts are also blocking (file io, gcs, s3 access), causing new requests
+  # but some parts are also blocking (some file io, gcs access), causing new requests
   # to wait despite idle cpu, if there are too few threads
   fork-join-executor {
-    parallelism-factor = 5.0 # Thread count = ceil(available processors * factor)
+    parallelism-factor = 10.0 # Thread count = ceil(available processors * factor)
     parallelism-min = 8 # Min number of threads to cap factor-based parallelism number to
-    parallelism-max = 300 # Max number of threads to cap factor-based parallelism number to
+    parallelism-max = 1000 # Max number of threads to cap factor-based parallelism number to
   }
 }