Skip to content

Commit cbe2e62

Browse files
authored
Merge branch 'master' into virtual-remote-datasets
2 parents 3a4b603 + 0863410 commit cbe2e62

File tree

5 files changed

+33
-23
lines changed

5 files changed

+33
-23
lines changed

docs/data/image_stacks.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ WEBKNOSSOS works with a wide range of modern bio-imaging formats and image stack
66
- [Multi layer file sequence](#multi-layer-image-file-sequence) containing multiple folders with image sequences that are interpreted as separate layers
77
- [Single-file images](#single-file-images) (OME-TIFF, TIFF, PNG, czi, raw, etc)
88

9-
Image stacks need to be converted to [WKW](./wkw.md) for WEBKNOSSOS. This happens automatically when using the web upload on [webknossos.org](https://webknossos.org) or can be done manually (see below).
9+
Image stacks need to be converted to [Zarr](./zarr.md) or [WKW](./wkw.md) datasets for WEBKNOSSOS. This happens automatically when using the web upload on [webknossos.org](https://webknossos.org) or can be done manually (see below).
1010

1111
## Single-Layer Image File Sequence
1212
When uploading multiple image files, these files are sorted numerically, and each one is interpreted as single section/slice within a 3D dataset.
@@ -60,7 +60,7 @@ You can manually convert image stacks through:
6060
### Conversion with CLI
6161
You can easily convert image stacks manually with the WEBKNOSSOS CLI.
6262
The CLI tool expects all image files in a single folder with numbered file names.
63-
After installing, you can convert image stacks to WKW datasets with the following command:
63+
After installing, you can convert image stacks to Zarr3 datasets with the following command:
6464

6565
```shell
6666
pip install webknossos
@@ -71,15 +71,15 @@ webknossos convert \
7171
data/source data/target
7272
```
7373

74-
This snippet converts an image stack that is located in directory called `data/source` into a WKW dataset which will be located at `data/target`.
74+
This snippet converts an image stack that is located in directory called `data/source` into a Zarr3 dataset which will be located at `data/target`.
7575
It will create a so called `color` layer containing your raw greyscale/color image.
7676
The supplied `--voxel-size` is specified in nanometers.
7777

7878
Read the full documentation at [WEBKNOSSOS CLI](https://docs.webknossos.org/cli).
7979

8080
### Conversion with Python
8181

82-
You can use the free [WEBKNOSSSO Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to WKW or integrate the conversion as part of an existing workflow.
82+
You can use the free [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to Zarr v3 or integrate the conversion as part of an existing workflow.
8383

8484
```python
8585
import webknossos as wk

docs/data/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ WEBKNOSSOS natively supports loading and streaming data in the following formats
1010
- [N5](./n5.md)
1111
- [Image Stacks (through Conversion)](./image_stacks.md)
1212

13-
The WEBKNOSSOS-wrap (WKW) container format is used for all internal voxel data representations - both for the raw (microscopy) image datasets and segmentations. Skeleton annotations are saved as NML files.
13+
The Zarr3 format is used for all internal voxel data representations - both for the raw (microscopy) image datasets and segmentations. Skeleton annotations are saved as NML files.
1414

15-
Any dataset uploaded to webknossos.org will automatically be converted to WKW on upload - given its source file format is supported by WEBKNOSSOS. Alternatively, you can manually convert your datasets using the [WEBKNOSSOS CLI tool](https://docs.webknossos.org/cli) or use a custom script based on the [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py/index.html).
15+
Any dataset uploaded to webknossos.org will automatically be converted to Zarr3 on upload - given its source file format is supported by WEBKNOSSOS. Alternatively, you can manually convert your datasets using the [WEBKNOSSOS CLI tool](https://docs.webknossos.org/cli) or use a custom script based on the [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py/index.html).
1616

1717
Read more about uploading and configuring datasets on the [datasets page](../datasets/settings.md).

docs/data/upload_ui.md

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@ The easiest way to get started with working on your datasets is through the WEBK
1111

1212
![youtube-video](https://www.youtube.com/embed/ZvUJrv86w8w?start=17)
1313

14-
Internally, WEBKNOSSOS uses the [WKW-format](./wkw.md) by default to display your data.
15-
If your data is already in WKW you can simply drag your folder (or zip archive of that folder) into the upload view.
14+
Internally, WEBKNOSSOS uses the [Zarr3](./zarr.md) format by default to display your data.
15+
If your data is already in a data format like [WKW](./wkw.md), [Zarr or Zarr3](./zarr.md) you can simply drag your folder (or zip archive of that folder) into the upload view.
1616

17-
If your data is not in WKW, you can either:
17+
If your data is not in WKW or Zarr format, you can either:
1818

1919
- upload the data in a supported file format and WEBKNOSSOS will automatically import or convert it ([webknossos.org](https://webknossos.org) only).
2020
Depending on the size of the dataset, the conversion will take some time.
@@ -34,4 +34,22 @@ In particular, the following file formats are supported for uploading (and conve
3434
- [Neuroglancer Precomputed datasets](./neuroglancer_precomputed.md)
3535
- [N5 datasets](./n5.md)
3636

37+
We support a variety of data types for the uploaded data. To make sure that your data can be uploaded to WEBKNOSSOS take a look into this table of supported data types for color and segmentation layers:
38+
39+
| dtype | Color Layers | Segmentation Layers |
40+
|------------|------------|------------|
41+
| uint8 |||
42+
| uint16 |||
43+
| uint24 rgb || does not apply |
44+
| uint32 |||
45+
| uint64 || (✓) [(til 2⁵³−1)](https://github.com/scalableminds/webknossos/issues/6921) |
46+
| | | |
47+
| int8 |||
48+
| int16 |||
49+
| int32 |||
50+
| int64 |||
51+
| | | |
52+
| float |||
53+
| double |||
54+
3755
Once the data is uploaded (and potentially converted), you can further configure a dataset's [Settings](../datasets/settings.md) and double-check layer properties, fine tune access rights & permissions, or set default values for rendering.

docs/data/wkw.md

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,11 @@
11
# WKW
22

33
[webknossos-wrap (WKW)](https://github.com/scalableminds/webknossos-wrap) is a format optimized for large datasets of 3D voxel imagery and supports compression, efficient cutouts, multi-channel, and several base datatypes.
4-
It works well for large datasets and is built with modern file systems in mind and drives the majority of WEBKNOSSOS datasets.
4+
It works well for large datasets and is built with modern file systems in mind and drives a lot of WEBKNOSSOS datasets.
55

6-
WKW is versatile in the image formats it can hold: Grayscale, Multi-Channel, Segmentation, RGB, as well as a range of data types (e.g., `uint8`, `uint16`, `float32`).
6+
WKW is versatile in the image formats it can hold: Grayscale, Multi-Channel, Segmentation, RGB, as well as a range of data types (e.g., `uint8`, `uint16`, `float32`).
77
Additionally, WKW supports compression for disk space efficiency.
88

9-
Each layer of a WKW dataset may contain one of the following:
10-
11-
* Grayscale data (8 Bit, 16 Bit, Float), also referred to as `color` data
12-
* RGB data (24 Bit)
13-
* Segmentation data (8 Bit, 16 Bit, 32 Bit)
14-
159
## Examples
1610

1711
You can try the WKW support with the following datasets. Upload them to WEBKNOSSOS using the [web uploader](./upload_ui.md):

docs/data/zarr.md

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
WEBKNOSSOS works great with [OME-Zarr datasets](https://ngff.openmicroscopy.org/latest/index.html), sometimes called next-generation file format (NGFF).
44

5-
We strongly believe in this community-driven, cloud-native data format for n-dimensional datasets. Zarr is a first-class citizen in WEBKNOSSOS and will likely replace [WKW](./wkw.md) long term.
5+
We strongly believe in this community-driven, cloud-native data format for n-dimensional datasets. Therefore, Zarr is the new default data format in WEBKNOSSOS and replaced the previous [WKW](./wkw.md) format.
66

77
Zarr datasets can both be uploaded to WEBKNOSSOS through the [web uploader](./upload_ui.md) or [streamed from a remote server or the cloud](./streaming.md). When streaming and using several layers, import the first Zarr group and then use the UI to add more URIs/groups.
88

@@ -121,23 +121,22 @@ webknossos convert \
121121
--layer-name em \
122122
--voxel-size 11.24,11.24,25 \
123123
--chunk-shape 64,64,64 \
124-
--data-format zarr \
125124
--jobs 4 \
126125
input.tif output.zarr
127126

128127
webknossos compress --jobs 4 output.zarr
129128
webknossos downsample --jobs 4 output.zarr
130129
```
131130

132-
This example will create an unsharded Zarr v2 dataset with a voxel size of (4,4,4) nm<sup>3</sup> and a chunk size of (64,64,64) voxel.
131+
This example will create a sharded Zarr v3 dataset with a voxel size of (11.24, 11.24, 25) nm<sup>3</sup> and a chunk size of (64,64,64) voxel.
133132
A maximum of 4 parallel jobs will be used to parallelize the conversion, compression and downsampling.
134-
Using the `--data-format zarr3` argument will produce sharded Zarr v3 datasets.
133+
Using the `--data-format zarr` argument will produce unsharded Zarr v2 datasets.
135134

136135
Read the full documentation at [WEBKNOSSOS CLI](https://docs.webknossos.org/cli).
137136

138137
### Conversion with Python
139138

140-
You can use the free [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to Zarr or integrate the conversion as part of an existing workflow.
139+
You can use the free [WEBKNOSSOS Python library](https://docs.webknossos.org/webknossos-py) to convert image stacks to Zarr3 or integrate the conversion as part of an existing workflow.
141140

142141
```python
143142
import webknossos as wk
@@ -150,7 +149,6 @@ def main() -> None:
150149
voxel_size=(11, 11, 11),
151150
layer_category=wk.COLOR_CATEGORY,
152151
compress=True,
153-
data_format=wk.Dataformat.Zarr
154152
)
155153

156154
print(f"Saved {dataset.name} at {dataset.path}.")

0 commit comments

Comments
 (0)