Add pyramids on cpu based on Jordao's code #12
Conversation
|
Hello @ieivanov . I am trying to switch to this branch and pip install biahub in the conda environment I created. I am getting the following error:
Could you please help? Thanks! Edit: Based on talking to @edyoshikun , these conflicts are resolved on main and he said the conflict can be resolved by merging main into this branch. |
|
@Soorya19Pradeep I think you should be able to install |
mattersoflight
changed the title
| ) | ||
|
|
||
| target_store = dataset[str(level)].tensorstore() | ||
| target_store[:].write(downsampled[:].read().result()).result() |
There was a problem hiding this comment.
Just as a heads up, I was having some pretty bad memory issues when I was generating the pyramid like this. Memory would spike massivley near the end of writing. Just putting some sort of loop on writing seemed to cap it. Here's what my code looks like here:
step = dst_ts.chunk_layout.write_chunk.shape[0]
for start in range(0, downsampled_ts.shape[0], step):
stop = min(start + step, downsampled_ts.shape[0])
dst_ts[start:stop].write(downsampled_ts[start:stop]).result()Though also would downsampled[:].read().result() read the entire source level into memory?
There was a problem hiding this comment.
I agree, tensorstore is pretty bad with memory efficiency. You probably need some looping that is chunk-aligned here as @ivirshup suggested. Also putting them in a transaction will help tensorstore schedule/consolidate the I/O.
There was a problem hiding this comment.
See also google/tensorstore#223
There was a problem hiding this comment.
@ziw-liu I get the impression tensorstore can be great with memory, just not from the highest level api and there are a lot of foot guns 😆
Also putting them in a transaction will help tensorstore schedule/consolidate the I/O.
I had tried this, but did not see an improvement. Does this work well in your hands? Would love a pointer to some code where this worked so I can give it another shot.
| def pyramid(fov_path: Path, levels: int, method: str) -> None: | ||
| """ | ||
| Create pyramid levels for a single field of view using tensorstore downsampling. | ||
|
|
||
| This function uses cascade downsampling, where each level is downsampled from | ||
| the previous level rather than from level 0. This avoids aliasing artifacts | ||
| and chunk boundary issues that occur with large downsample factors. |
There was a problem hiding this comment.
Could this function move into iohub, say as a method of Position?
6 participants
This PR largely copies https://github.com/royerlab/imgproc-monorepo/blob/main/impp/impp/pyramid.py expect that pyramids are implemented in CPU rather than GPU as stitches of OPS datasets do not easily fit in GPU memory. We should discuss how we can reduce down to one version of pyramiding functionality - maybe move that to
iohub?TODO:
cc @ziw-liu @JoOkuma