Missing dependency

[araikes]

Hi @TheChymera

I was trying to build this into a Singularity container and got... almost there. I get an error because the Science overlay no longer has scikit-image.

Any thoughts on a solution for that?

[TheChymera]

Hi @araikes , happy to hear you're interested to use SAMRI.

dev-python/scikit-image is now in ::gentoo, which is to say the main Gentoo package repository, so in a sense it's even more accessible now :)

The issue you have encountered is not caused by it being moved, but rather by it being renamed from sci-libs/scikits_image to dev-python/scikit-image, which I have now fixed in this commit.

If you are looking into containerization, we are currently overhauling the .gentoo specification.
Basically we have a container file model which will generate an image based on a .gentoo directory, used:

• For an article
• For an ongoing research project
• For an atlas standardization workflow

We are using SAMRI images here, and we have them for both:

• Singularity: https://gin.g-node.org/asmacdo/opfvta-singularity
• OCI (Docker/Podman): https://hub.docker.com/r/centerforopenneuroscience/opfvta

The spec is not updated in the SAMRI package, because thus far we didn't need it on its own, but we can prioritize this it would be helpful for you. Or if you want, you could also just submit an analogous PR, and I'd be happy to review and fix it up.

[araikes]

@TheChymera,

That's terrific and I'd rather not rebuild the wheel if I don't need to. In essence, I'm looking to test alternative preprocessing pipelines for my mouse data (particularly T2-weighted anatomical images) as masking is inconsistent at best. I assume that the SAMRI images would suit my needs at least for the time being, though I'm interested in the atlas standardization workflow, as the relationship between diffusion-based tractography and ABI data would be interesting.

I'll give the existing images a shot and see if they give me any issues

Thanks

[TheChymera]

Let me know. As for the ABI/DTI idea, feel free to reach out via email, I'd also be interested in looking at that. I actually expect there to be significant differences, since DTI is insensitive to low-density projections in areas of low fiber coherence. So you tend to not see neuromodulatory networks, which are the most interesting part in tracing studies. In a sense a nuanced negative finding (i.e. these are the things you're misisng in DTI) might be even more interesting.

[araikes]

If I don't have functional images, will SAMRI fail on me?

singularity run $SAMRI SAMRI generic-prep nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects sub-1024 -o $PWD/derivatives/samri
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'type'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 743, in common_select
    functional_scan_types = data_selection.loc[data_selection['type'] == 'func']['acq'].values
  File "/usr/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'type'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'datatype'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 392, in generic
    bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind = common_select(
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 746, in common_select
    functional_scan_types = data_selection.loc[data_selection['datatype'] == 'func']['acquisition'].values
  File "/usr/lib/python3.10/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'datatype'

[TheChymera]

Short answer, no, it should not fail.
The SAMRI data preparation functions can do functional, structural, or both.
The issue is probably that the logic fails to detect it's just structural. Let me look into it. Are you using the opfvta-reexecution-article image?

[TheChymera]

Ok, so the code that originally handled the struct-only processing has degraded over time since I don't use it, and we don't yet have a robust test suite.
In any case I fixed it in 1f14439 .
Thanks for reporting this.

One thing to note is that in addition to the issue in SAMRI, there's another issue in your invocation.
The --subjects parameter takes the subject values, not including the sub- key.
A correct invocation would be, e.g.:

(dev) [deco]/tmp ❱ SAMRI generic-prep -m '/usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii' --subjects 6471 -s '{"acquisition":["TurboRARE"]}' -o . /usr/share/opfvta_bidsdata/ '/usr/share/mouse-brain-templates/dsurqec_200micron.nii'

The above command now works on my machine, let me know if it doesn't on yours.
If it fails for you but not on the reference data, please send me a link to your data so I can check whether the problem might be input data formatting.

[araikes]

I was running the image up on DockerHub. If I rebuild the image using the Dockerfile for e.g. the article, will it pull the current version with that fix in place?

I also assume that it's going to have some issues. I used brkraw, rather than SAMRI, to convert to BIDS, so my files are named as sub-XXX_ses-XX_T2w.nii.gz rather than having the acq- tag. Based on some of the documentation, it looks like I should be able to use -s '{"modality":["T2w"]}', yes?

[TheChymera]

will it pull the current version with that fix in place?

I'm actually not sure, I think it uses the last versioned release from portage. Perhaps I should set up a container here for more flexibility.

Based on some of the documentation, it looks like I should be able to use -s '{"modality":["T2w"]}', yes?

Yes, tested it here with my data as well just to be sure, and that works.

[araikes]

Glad that works on your end.

If you come up with a container solution, let me know. I don't know Gentoo enough to modify any of the Dockerfiles to make sure that its grabbing the current state of the Git repo.

Or, alternatively, do you know where the .py files are in the filesystem? I could just bind the new version over the top of that path.

[araikes]

Last note... for now....

If I fix the -s parameter to match my modality, the current Docker container throws an error:

singularity run --containall -B $PWD:$PWD /groups/adamraikes/singularity_images/opfvta_2.0.0-alpha.sif SAMRI generic-prep $PWD/nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 1024 -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'

Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
usage: SAMRI generic-prep [-h] [-a] [-d] [--functional-blur-xy] [-f FUNCTIONAL_MATCH] [--functional-registration-method FUNCTIONAL_REGISTRATION_METHOD] [-k] [--n-jobs] [--n-jobs-percentage N_JOBS_PERCENTAGE] [-o OUT_BASE] [--realign REALIGN] [-m REGISTRATION_MASK] [--sessions SESSIONS] [-s STRUCTURAL_MATCH]
                          [--subjects SUBJECTS] [--tr TR] [-w WORKFLOW_NAME] [--params PARAMS] [--phase-dictionary PHASE_DICTIONARY] [--enforce-dummy-scans ENFORCE_DUMMY_SCANS] [--exclude EXCLUDE]
                          bids_base template
SAMRI generic-prep: error: argument -s/--structural-match: invalid loads value: '{modality:[T2w]}'

[TheChymera]

do you know where the .py files are in the filesystem? I could just bind the new version over the top of that path.

So the files that were updated are the ones in the last commits, in the image they reside under /usr/lib/python3.11/site-packages/samri.

[...] -s paramater [...]

I have no idea why you might be getting that error, I can't reproduce it here on my end. The analogous command to yours, with my data, is the following:

(dev) [deco]/tmp ❱ SAMRI generic-prep /usr/share/opfvta_bidsdata  /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 6471 -o $PWD/ -s '{"modality":["T2w"]}'
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.11/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.11/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
/home/chymera/src/SAMRI/samri/pipelines/utils.py:235: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '['0' '0' '1' '1' '1' '0' '0' '0' '0' '1' '0' '0' '0' '1' '1' '0' '1' '1'
 '0' '0' '1' '0' '0' '0' '1' '1' '1' '0' '1' '0' '1' '1' '0' '0' '0' '1'
 '0' '1' '1' '0' '1' '0' '1' '1' '0' '0' '0' '1' '1' '0' '1' '0' '0' '0'
 '1' '1' '1' '0' '0' '0' '0' '1' '0' '1' '1' '0' '0' '0' '1' '1' '1' '0'
 '0' '1' '0' '1' '1' '0' '0' '0' '0' '0' '0' '0' '0' '0' '0' '1' '1' '0'
 '0' '1' '0' '0' '0' '0' '0' '0' '0' '0' '1' '0' '1' '1' '0' '0' '0' '0'
 '0' '0' '0' '1' '1' '0' '1' '0' '0' '0' '0' '0' '1' '1' '1' '0' '0' '0'
 '0' '0' '0' '0' '1' '1' '0' '0' '1' '0' '0' '0' '0' '0' '0' '0' '1' '0'
 '1' '1' '0' '1' '0' '1' '0' '1' '0' '0' '1' '1' '0' '0' '1' '0' '1' '1'
 '0' '0' '1' '0' '0' '0' '0' '0' '0' '0' '1' '0' '1' '0' '1' '0' '0' '0'
 '1' '1' '1' '0' '0' '0' '1' '0' '0' '1' '0' '1' '0' '0' '0' '1' '0' '0'
 '1' '0' '1' '1' '0' '0' '1' '1' '0' '0' '1' '1' '1' '0' '0' '1' '1' '0'
 '1' '0' '0' '0' '0' '0' '0' '0' '0' '1' '0' '1' '0' '0' '1' '0' '0' '0'
 '0' '0' '0' '0' '0' '0' '0' '0' '0' '0' '0' '0']' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
  df.loc[df['run'].notna(),'run'] = df.loc[df['run'].notnull(),'run'].apply(int).apply(str)
                                                 path subject session task acquisition        acq  run modality  type scan_type
26  /usr/share/opfvta_bidsdata/sub-6471/ses-ofM/an...    6471     ofM  NaN   TurboRARE  TurboRARE  NaN      T2w  anat
27  /usr/share/opfvta_bidsdata/sub-6471/ses-ofMr1/...    6471   ofMr1  NaN   TurboRARE  TurboRARE  NaN      T2w  anat
[['6471', 'ofM'], ['6471', 'ofMr1']]
240215-21:27:28,150 nipype.workflow INFO:
	 Generated workflow graph: /tmp/generic_work/graph.png (graph2use=hierarchical, simple_form=True).
240215-21:27:28,167 nipype.workflow INFO:
	 Workflow generic_work settings: ['check', 'execution', 'logging', 'monitoring']
240215-21:27:28,179 nipype.workflow INFO:
	 Running in parallel.
240215-21:27:28,181 nipype.workflow INFO:
	 [MultiProc] Running 0 tasks, and 2 jobs ready. Free memory (GB): 56.27/56.27, Free processors: 10/10.
240215-21:27:28,249 nipype.workflow INFO:
	 [Job 0] Cached (generic_work.get_s_scan).
240215-21:27:28,260 nipype.workflow INFO:
	 [Job 1] Cached (generic_work.get_s_scan).

It also doesn't sound like it's related to the data since it pops up before any read-in is attempted. The weird thing is that the strings in the variable aren't quoted in the error message '{modality:[T2w]}'... might be an indicator. Perhaps something is wrong with your shell?

I could get a similar error with a slightly modified call.
The difference is that here, instead of '{"modality":["T2w"]}', I use "{"modality":["T2w"]}" (the unquoted output gave me the idea to try this out).

(dev) [deco]/tmp ❱ SAMRI generic-prep /usr/share/opfvta_bidsdata  /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 6471 -o $PWD/ -s "{"modality":["T2w"]}"
Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.11/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.11/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
usage: SAMRI generic-prep [-h] [-a] [-d] [--functional-blur-xy] [--functional-match FUNCTIONAL_MATCH] [--functional-registration-method FUNCTIONAL_REGISTRATION_METHOD] [-k] [--n-jobs] [--n-jobs-percentage N_JOBS_PERCENTAGE] [-o OUT_BASE] [--realign REALIGN]
                          [--registration-mask REGISTRATION_MASK] [--sessions [SESSIONS ...]] [--structural-match STRUCTURAL_MATCH] [--subjects [SUBJECTS ...]] [-t TR] [-w WORKFLOW_NAME] [--params PARAMS] [--phase-dictionary PHASE_DICTIONARY]
                          [--enforce-dummy-scans ENFORCE_DUMMY_SCANS] [--exclude EXCLUDE]
                          bids-base template
SAMRI generic-prep: error: argument --structural-match/-s: invalid loads value: '{modality:[T2w]}'

Looks like somewhere in between the input and samri, in your case, the single quotes get converted to double quotes. Maybe it's singularity?

[araikes]

Found it. It's in python3.10 not python3.11 and I was able to start the process by opening a shell (rather than directly running it). E.g. I changed from:

singularity run --containall -B $PWD:$PWD -B /groups/adamraikes/git_repos/SAMRI:/usr/lib/python3.10/site-packages/samri /groups/adamraikes/singularity_images/opfvta_2.0.0-alpha.sif SAMRI generic-prep $PWD/nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 1024 -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'

to

singularity shell --containall -B $PWD:$PWD -B /groups/adamraikes/git_repos/SAMRI/samri:/usr/lib/python3.10/site-packages/samri /groups/adamraikes/singularity_images/opfvta_2.0.0-alpha.sif

SAMRI generic-prep $PWD/nifti /usr/share/mouse-brain-templates/dsurqec_200micron.nii -m /usr/share/mouse-brain-templates/dsurqec_200micron_mask.nii --subjects 1024 -o $PWD/derivatives/samri -s '{"modality":["T2w"]}'

However, I now get the following:

Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 392, in generic
    bids_base, out_base, out_dir, template, registration_mask, data_selection, functional_scan_types, structural_scan_types, subjects_sessions, func_ind, struct_ind = common_select(
  File "/usr/lib/python3.10/site-packages/samri/pipelines/preprocess.py", line 769, in common_select
    subjects_sessions = data_selection[["subject","session"]].drop_duplicates().values.tolist()
  File "/usr/lib/python3.10/site-packages/pandas/core/frame.py", line 3813, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/usr/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['subject', 'session'], dtype='object')] are in the [columns]"

Further, if I try SAMRI diagnose, I get

Failed to import duecredit due to No module named 'duecredit'
/usr/lib/python3.10/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/lib/python3.10/site-packages/nilearn/input_data/__init__.py:27: FutureWarning: The import path 'nilearn.input_data' is deprecated in version 0.9. Importing from 'nilearn.input_data' will be possible at least until release 0.13.0. Please import from 'nilearn.maskers' instead.
  warnings.warn(message, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.10/SAMRI", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/samri/cli.py", line 13, in main
    argh.dispatch_commands([diagnose, bru2bids, l1, generic, legacy])
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 349, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 179, in dispatch
    for line in lines:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 290, in _execute_command
    for line in result:
  File "/usr/lib/python3.10/site-packages/argh/dispatching.py", line 273, in _call
    result = function(*positional, **keywords)
  File "/usr/lib/python3.10/site-packages/samri/pipelines/diagnostics.py", line 84, in diagnose
    datafind = nio.DataFinder()
  File "/usr/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 189, in __init__
    BaseInterface._etelemetry_version_data = check_latest_version() or "n/a"
  File "/usr/lib/python3.10/site-packages/nipype/__init__.py", line 85, in check_latest_version
    import etelemetry
ModuleNotFoundError: No module named 'etelemetry'

As an aside, I pass BIDS validation with ignorable issues related to TSV files and inconsistent session numbers.

[araikes]

Manually check: None of the nipype imports work due to missing etelemetry and can confirm that it's not present in /usr/lib/python3.10/site-packages

[TheChymera]

It's not, but it should be patched to not give the error, let me check why it's giving you the error.