Merge branch 'master' into account-settings-page

Author: hotzenklotz

docs/data/image_stacks.md

@@ -6,7 +6,7 @@ WEBKNOSSOS works with a wide range of modern bio-imaging formats and image stack
 - [Multi layer file sequence](#multi-layer-image-file-sequence) containing multiple folders with image sequences that are interpreted as separate layers
 - [Single-file images](#single-file-images) (OME-TIFF, TIFF, PNG, czi, raw, etc)
 
-Image stacks need to be converted to [WKW](./wkw.md) for WEBKNOSSOS. This happens automatically when using the web upload on [webknossos.org](https://webknossos.org) or can be done manually (see below).
+Image stacks need to be converted to [Zarr](./zarr.md) or [WKW](./wkw.md) datasets for WEBKNOSSOS. This happens automatically when using the web upload on [webknossos.org](https://webknossos.org) or can be done manually (see below).
 
 ## Single-Layer Image File Sequence
 When uploading multiple image files, these files are sorted numerically, and each one is interpreted as single section/slice within a 3D dataset.
@@ -60,7 +60,7 @@ You can manually convert image stacks through:
 ### Conversion with CLI
 You can easily convert image stacks manually with the WEBKNOSSOS CLI.
 The CLI tool expects all image files in a single folder with numbered file names.
-After installing, you can convert image stacks to WKW datasets with the following command:
+After installing, you can convert image stacks to Zarr3 datasets with the following command:
 
 ```shell
 pip install webknossos
@@ -71,15 +71,15 @@ webknossos convert \
   data/source data/target
 ```
 
-This snippet converts an image stack that is located in directory called `data/source` into a WKW dataset which will be located at `data/target`.
+This snippet converts an image stack that is located in directory called `data/source` into a Zarr3 dataset which will be located at `data/target`.
 It will create a so called `color` layer containing your raw greyscale/color image.
 The supplied `--voxel-size` is specified in nanometers.
 
 Read the full documentation at [WEBKNOSSOS CLI](https://docs.webknossos.org/cli).
 
 ### Conversion with Python
 
-You can use the free [WEBKNOSSSO Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to WKW or integrate the conversion as part of an existing workflow. 
+You can use the free [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to Zarr v3 or integrate the conversion as part of an existing workflow. 
 
 ```python
 import webknossos as wk

docs/data/index.md

@@ -10,8 +10,8 @@ WEBKNOSSOS natively supports loading and streaming data in the following formats
 - [N5](./n5.md)
 - [Image Stacks (through Conversion)](./image_stacks.md)
 
-The WEBKNOSSOS-wrap (WKW) container format is used for all internal voxel data representations - both for the raw (microscopy) image datasets and segmentations. Skeleton annotations are saved as NML files. 
+The Zarr3 format is used for all internal voxel data representations - both for the raw (microscopy) image datasets and segmentations. Skeleton annotations are saved as NML files. 
 
-Any dataset uploaded to webknossos.org will automatically be converted to WKW on upload - given its source file format is supported by WEBKNOSSOS. Alternatively, you can manually convert your datasets using the [WEBKNOSSOS CLI tool](https://docs.webknossos.org/cli) or use a custom script based on the [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py/index.html).
+Any dataset uploaded to webknossos.org will automatically be converted to Zarr3 on upload - given its source file format is supported by WEBKNOSSOS. Alternatively, you can manually convert your datasets using the [WEBKNOSSOS CLI tool](https://docs.webknossos.org/cli) or use a custom script based on the [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py/index.html).
 
 Read more about uploading and configuring datasets on the [datasets page](../datasets/settings.md).

docs/data/upload_ui.md

@@ -11,10 +11,10 @@ The easiest way to get started with working on your datasets is through the WEBK
 
 ![youtube-video](https://www.youtube.com/embed/ZvUJrv86w8w?start=17)
 
-Internally, WEBKNOSSOS uses the [WKW-format](./wkw.md) by default to display your data.
-If your data is already in WKW you can simply drag your folder (or zip archive of that folder) into the upload view.
+Internally, WEBKNOSSOS uses the [Zarr3](./zarr.md) format by default to display your data.
+If your data is already in a data format like [WKW](./wkw.md), [Zarr or Zarr3](./zarr.md) you can simply drag your folder (or zip archive of that folder) into the upload view.
 
-If your data is not in WKW, you can either:
+If your data is not in WKW or Zarr format, you can either:
 
 - upload the data in a supported file format and WEBKNOSSOS will automatically import or convert it ([webknossos.org](https://webknossos.org) only). 
 Depending on the size of the dataset, the conversion will take some time. 
@@ -34,4 +34,22 @@ In particular, the following file formats are supported for uploading (and conve
 - [Neuroglancer Precomputed datasets](./neuroglancer_precomputed.md)
 - [N5 datasets](./n5.md)
 
+We support a variety of data types for the uploaded data. To make sure that your data can be uploaded to WEBKNOSSOS take a look into this table of supported data types for color and segmentation layers:
+
+|   dtype    | Color Layers  |       Segmentation Layers   |
+|------------|------------|------------|
+|   uint8    |       ✓     |   ✓                       |
+|   uint16   |       ✓     |   ✓                       |
+|   uint24  rgb  |       ✓   |   does not apply        |
+|   uint32   |       ✓     |   ✓                       |
+|   uint64   |       ✗     |   (✓)  [(til 2⁵³−1)](https://github.com/scalableminds/webknossos/issues/6921)          |
+|   |  |  |
+|    int8    |       ✓     |        ✓                   |
+|    int16   |       ✓     |        ✓                   |
+|    int32   |       ✓     |        ✓                   |
+|    int64   |       ✗     |        ✓                   |
+|   |  |  |
+|    float   |       ✓     |        ✗                   |
+|    double  |       ✗     |        ✗                   |
+
 Once the data is uploaded (and potentially converted), you can further configure a dataset's [Settings](../datasets/settings.md) and double-check layer properties, fine tune access rights & permissions, or set default values for rendering.

docs/data/wkw.md

@@ -1,17 +1,11 @@
 # WKW
 
 [webknossos-wrap (WKW)](https://github.com/scalableminds/webknossos-wrap) is a format optimized for large datasets of 3D voxel imagery and supports compression, efficient cutouts, multi-channel, and several base datatypes.
-It works well for large datasets and is built with modern file systems in mind and drives the majority of WEBKNOSSOS datasets.
+It works well for large datasets and is built with modern file systems in mind and drives a lot of WEBKNOSSOS datasets.
 
-WKW is versatile in the image formats it can hold: Grayscale, Multi-Channel, Segmentation, RGB, as well as a range of data types (e.g., `uint8`,  `uint16`, `float32`).
+WKW is versatile in the image formats it can hold: Grayscale, Multi-Channel, Segmentation, RGB, as well as a range of data types (e.g., `uint8`, `uint16`, `float32`).
 Additionally, WKW supports compression for disk space efficiency.
 
-Each layer of a WKW dataset may contain one of the following:
-
-* Grayscale data (8 Bit, 16 Bit, Float), also referred to as `color` data
-* RGB data (24 Bit)
-* Segmentation data (8 Bit, 16 Bit, 32 Bit)
-
 ## Examples
 
 You can try the WKW support with the following datasets. Upload them to WEBKNOSSOS using the [web uploader](./upload_ui.md): 

docs/data/zarr.md

@@ -2,7 +2,7 @@
 
 WEBKNOSSOS works great with [OME-Zarr datasets](https://ngff.openmicroscopy.org/latest/index.html), sometimes called next-generation file format (NGFF).
 
-We strongly believe in this community-driven, cloud-native data format for n-dimensional datasets. Zarr is a first-class citizen in WEBKNOSSOS and will likely replace [WKW](./wkw.md) long term.
+We strongly believe in this community-driven, cloud-native data format for n-dimensional datasets. Therefore, Zarr is the new default data format in WEBKNOSSOS and replaced the previous [WKW](./wkw.md) format.
 
 Zarr datasets can both be uploaded to WEBKNOSSOS through the [web uploader](./upload_ui.md) or [streamed from a remote server or the cloud](./streaming.md). When streaming and using several layers, import the first Zarr group and then use the UI to add more URIs/groups.
 
@@ -121,23 +121,22 @@ webknossos convert \
   --layer-name em \
   --voxel-size 11.24,11.24,25 \
   --chunk-shape 64,64,64 \
-  --data-format zarr \
   --jobs 4 \
   input.tif output.zarr
 
 webknossos compress --jobs 4 output.zarr
 webknossos downsample --jobs 4 output.zarr
 ```
 
-This example will create an unsharded Zarr v2 dataset with a voxel size of (4,4,4) nm<sup>3</sup> and a chunk size of (64,64,64) voxel. 
+This example will create a sharded Zarr v3 dataset with a voxel size of (11.24, 11.24, 25) nm<sup>3</sup> and a chunk size of (64,64,64) voxel. 
 A maximum of 4 parallel jobs will be used to parallelize the conversion, compression and downsampling.
-Using the `--data-format zarr3` argument will produce sharded Zarr v3 datasets.
+Using the `--data-format zarr` argument will produce unsharded Zarr v2 datasets.
 
 Read the full documentation at [WEBKNOSSOS CLI](https://docs.webknossos.org/cli).
 
 ### Conversion with Python
 
-You can use the free [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to Zarr or integrate the conversion as part of an existing workflow. 
+You can use the free [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to Zarr3 or integrate the conversion as part of an existing workflow. 
 
 ```python
 import webknossos as wk
@@ -150,7 +149,6 @@ def main() -> None:
         voxel_size=(11, 11, 11),
         layer_category=wk.COLOR_CATEGORY,
         compress=True,
-        data_format=wk.Dataformat.Zarr
     )
 
     print(f"Saved {dataset.name} at {dataset.path}.")

frontend/javascripts/admin/rest_api.ts

@@ -1068,10 +1068,12 @@ export function getDatasetDefaultConfiguration(datasetId: string): Promise<Datas
 export function updateDatasetDefaultConfiguration(
   datasetId: string,
   datasetConfiguration: DatasetConfiguration,
+  options?: RequestOptions,
 ): Promise<ArbitraryObject> {
   return Request.sendJSONReceiveJSON(`/api/datasetConfigurations/default/${datasetId}`, {
     method: "PUT",
     data: datasetConfiguration,
+    ...options,
   });
 }
 

frontend/javascripts/components/brain_spinner.tsx

@@ -75,7 +75,7 @@ export function BrainSpinnerWithError({
     <BrainSpinner
       message={
         <>
-          <div style={{ textAlign: "center" }}>
+          <div className="initialization-error-message" style={{ textAlign: "center" }}>
             {gotUnhandledError ? messages["tracing.unhandled_initialization_error"] : message}
           </div>
           <div className="flex-center-child" style={{ gap: 8 }}>

frontend/javascripts/libs/browser_feature_check.tsx

@@ -11,6 +11,7 @@ export default function checkBrowserFeatures() {
     new Map([[1, 2]]).values().map((v) => v);
     [].at(0);
     new Set().difference(new Set());
+    [].values().flatMap((el) => [el, el]);
   } catch (exception) {
     Toast.warning(
       <div>

frontend/javascripts/main.tsx

@@ -86,7 +86,6 @@ async function loadOrganization() {
     Store.dispatch(setActiveOrganizationAction(organization));
   }
 }
-
 document.addEventListener("DOMContentLoaded", async () => {
   ErrorHandling.initialize({
     throwAssertions: false,

frontend/javascripts/test/puppeteer/dataset_rendering.wkorg_screenshot.ts

@@ -1,6 +1,7 @@
 import path from "node:path";
 import { compareScreenshot, isPixelEquivalent } from "./screenshot_helpers";
 import {
+  getDefaultRequestOptions,
   getNewPage,
   screenshotDatasetView,
   type ScreenshotTestContext,
@@ -10,6 +11,9 @@ import {
 } from "./dataset_rendering_helpers";
 import { encodeUrlHash } from "viewer/controller/url_manager";
 import { describe, it, beforeEach, afterEach, expect } from "vitest";
+import { updateDatasetDefaultConfiguration } from "admin/rest_api";
+import type { DatasetConfiguration } from "viewer/store";
+import type { BLEND_MODES } from "viewer/constants";
 
 process.on("unhandledRejection", (err, promise) => {
   console.error("Unhandled rejection (promise: ", promise, ", reason: ", err, ").");
@@ -42,7 +46,59 @@ const viewOverrides: Record<string, string> = {
   ),
 };
 
-describe("WebKnossos.org Dataset Rendering", () => {
+// The following configuration will be set during the test.
+// The tested dataset is *the* example dataset for WK which users
+// are guided to from the landing page. We want to ensure good defaults
+// and in case they were changed accidentally, it is good that they are
+// reset automatically each night.
+// If we want different defaults, they should be set here.
+const datasetConfiguration: DatasetConfiguration = {
+  layers: {
+    color: {
+      alpha: 100,
+      gammaCorrectionValue: 1,
+      min: 0,
+      color: [255, 255, 255],
+      max: 255,
+      isInverted: false,
+      isInEditMode: false,
+      isDisabled: false,
+      intensityRange: [74, 165],
+    },
+    predictions: {
+      alpha: 50,
+      gammaCorrectionValue: 1,
+      min: 0,
+      color: [255, 255, 255],
+      max: 255,
+      isInverted: false,
+      isInEditMode: false,
+      isDisabled: true,
+      intensityRange: [103, 199],
+    },
+    segmentation: {
+      alpha: 20,
+      gammaCorrectionValue: 1,
+      mapping: null,
+      color: [255, 255, 255],
+      isInverted: false,
+      isInEditMode: false,
+      isDisabled: false,
+    },
+  },
+  fourBit: false,
+  interpolation: false,
+  segmentationPatternOpacity: 25,
+  colorLayerOrder: ["color", "predictions"],
+  position: [2924, 4474, 1770],
+  zoom: 0.9,
+  blendMode: "Additive" as BLEND_MODES,
+  nativelyRenderedLayerName: null,
+  loadingStrategy: "PROGRESSIVE_QUALITY",
+  renderMissingDataBlack: true,
+};
+
+describe("webknossos.org Dataset Rendering", () => {
   beforeEach<ScreenshotTestContext>(async (context) => {
     await setupBeforeEach(context);
   });
@@ -51,42 +107,47 @@ describe("WebKnossos.org Dataset Rendering", () => {
     await setupAfterEach(context);
   });
 
-  it.sequential<ScreenshotTestContext>(
-    `should render dataset ${demoDatasetName} correctly`,
-    async ({ browser }) => {
-      await withRetry(
-        3,
-        async () => {
-          const response = await fetch(
-            `${URL}/api/datasets/disambiguate/${owningOrganization}/${demoDatasetName}/toId`,
-          );
-          const { id: datasetId } = await response.json();
+  it<ScreenshotTestContext>(`should render dataset ${demoDatasetName} correctly`, async ({
+    browser,
+  }) => {
+    await withRetry(
+      3,
+      async () => {
+        const response = await fetch(
+          `${URL}/api/datasets/disambiguate/${owningOrganization}/${demoDatasetName}/toId`,
+        );
+        const { id: datasetId } = await response.json();
 
-          const page = await getNewPage(browser);
-          const { screenshot, width, height } = await screenshotDatasetView(
-            page,
-            URL,
-            datasetId,
-            viewOverrides[demoDatasetName],
-          );
-          const changedPixels = await compareScreenshot(
-            screenshot,
-            width,
-            height,
-            SCREENSHOTS_BASE_PATH,
-            demoDatasetName,
-          );
-          await page.close();
+        await updateDatasetDefaultConfiguration(
+          datasetId,
+          datasetConfiguration,
+          getDefaultRequestOptions(URL),
+        );
 
-          return isPixelEquivalent(changedPixels, width, height);
-        },
-        (condition) => {
-          expect(
-            condition,
-            `Dataset with name: "${demoDatasetName}" does not look the same, see ${demoDatasetName}.diff.png for the difference and ${demoDatasetName}.new.png for the new screenshot.`,
-          ).toBe(true);
-        },
-      );
-    },
-  );
+        const page = await getNewPage(browser);
+        const { screenshot, width, height } = await screenshotDatasetView(
+          page,
+          URL,
+          datasetId,
+          viewOverrides[demoDatasetName],
+        );
+        const changedPixels = await compareScreenshot(
+          screenshot,
+          width,
+          height,
+          SCREENSHOTS_BASE_PATH,
+          demoDatasetName,
+        );
+        await page.close();
+
+        return isPixelEquivalent(changedPixels, width, height);
+      },
+      (condition) => {
+        expect(
+          condition,
+          `Dataset with name: "${demoDatasetName}" does not look the same, see ${demoDatasetName}.diff.png for the difference and ${demoDatasetName}.new.png for the new screenshot.`,
+        ).toBe(true);
+      },
+    );
+  });
 });