LORIS-MRI Release v22.0.3

This bug fix release resolves an issue for the insertion of the DWI derived images (color FA, tensor etc...) in the tarchive_series. Because those series does not contain any echo time, repetition or inversion time in their headers, the insert statement into the tarchive_series fails. Before release 22.0.0, this failed silently but since 22.0.0, any failed insert statement at the dicomTar.pl level will stop the insertion pipeline. The changes in this bug fix will resolve that issue for those particular series. See #568

LORIS-MRI Release v23.0.2

This is a bug fix release for LORIS-MRI version 23.0.0.

It resolves an issue where __MACOSX directories could not be deleted from the uploaded DICOMs as the directory was not empty. (See PR #566 for more details).

LORIS-MRI Release v23.0.0

This release of the LORIS-MRI repository accompanies the LORIS 23.0.0 release.

• List of all LORIS-MRI 23.0.0 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• Add support for multiple MRI protocols based on projects, subprojects or visit labels (#369) Feel free to read the updated documentation in the LORIS-MRI docs directory for full documentation on how to set protocol based on a project, subproject or visit label. Instructions on how to set that up:

  • First you have to run the SQL patch that updates the database schema for this new feature. Once you have done this, the behaviour of the pipeline should not have changed: all the lines of the mri_protocol and mri_protocol_checks will be active for all scans . To create a new MRI protocol group:
    1. Insert a record in mri_protocol_group with the new group name and ID.
    2. Associate each line in table mri_protocol to a protocol group in mri_protocol_group by filling out columnMriProtocolGroupID.
    3. Determine the conditions under which the new group will be used to identify the type of a scan. Conditions include the project ID, sub-project ID or visit label of the session at which the scan was done. There are columns in table mri_protocol_group_target for each of these types of conditions. When more than one column is filled in this table, the conditions are ANDed together.
  • The steps to create a new MRI protocol checks group are analog to those described above but use tables mri_protocol_checks_group and mri_protocol_checks_group_target instead.
  • Each line of the mri_protocol table belongs to either one group or no group at all, and this will be enforced in the MySQL schema. Lines of mri_protocol_checks can belong to 0, 1 or many protocol checks group though.

• Addition of a mass NIfTI pic creation script for the python pipeline (#515)

• Creation of a DICOM archive validation script for the python pipeline (#517)

Improvements

• Allow deletion of a file with a specific base name in delete_imaging_upload.pl (#508)

• Add the $subjectID{'isPhantom'} key in the getSubjectIDs function of the profileTemplate.pl (#516)

• Fixed the way the value was encoded in the events file for electrophysiology datasets (#521)

• Fixed a "Can't call method 'prepare' on unblessed reference" message when the pipeline was running (#533)

• Fixed errors that were displayed when running the LORIS-MRI Perl pipeline on Mac OS X (#534)

• Fixed a bug in MakeArchiveLocationRelative.pl tool script (#537)

• Fixed the invalid find command in script dicomTar.pl that fails to delete files with name __MACOSX (#540)

• Fixed a bug in the mass_electrophysiology_chunking script (#543)

• Fixed the timestamp not being populated in the parameter_file table when inserting into that table from the python pipeline (#544)

Clean up

• Deleted unused setMRISession function from MRIProcessingUtility.pm (#518)

• Removed leftover occurrences of requiresStaging (#521)

Notes for Existing Projects

• Ensure that the getSubjectIDs function in your prod config file contains the changes from PR #516

• If you are using the python pipelines, ensure to update your database_config.py config file with the following content (#517):

import re
from lib.imaging import Imaging

def get_subject_ids(db, dicom_value=None, scanner_id=None):

    subject_id_dict = {}

    imaging = Imaging(db, False)

    phantom_match   = re.search('(pha)|(test)', dicom_value, re.IGNORECASE)
    candidate_match = re.search('([^_]+)_(\d+)_([^_]+)', dicom_value, re.IGNORECASE)

    if phantom_match:
        subject_id_dict['isPhantom']  = True
        subject_id_dict['CandID']     = imaging.get_scanner_candid(scanner_id)
        subject_id_dict['visitLabel'] = dicom_value.strip()
        subject_id_dict['createVisitLabel'] = 1
    elif candidate_match:
        subject_id_dict['isPhantom']  = False
        subject_id_dict['PSCID']      = candidate_match.group(1)
        subject_id_dict['CandID']     = candidate_match.group(2)
        subject_id_dict['visitLabel'] = candidate_match.group(3)
        subject_id_dict['createVisitLabel'] = 0

    return subject_id_dict

• Caveat related to the support for multiple MRI protocols:
  • Ensure to run the patch on the LORIS side to populate the table changes made to the MRI protocol tables for your project. Information about the new table structures can be found in the LORIS-MRI docs directory here. Note that by default, all entries in mri_protocol and mri_protocol_checks will by default be assigned a Default MRI protocol group and Default MRI protocol checks group that will apply to all your projects, subprojects and visit labels.

LORIS-MRI Release v22.0.1

This release of the LORIS-MRI repository accompanies the LORIS 22.0.1 release.

This release fixes a bug in the python BIDS insertion scripts. The regular expression to determine a file type is less permissive than before and does only check the extension of the file now instead of the whole path. See #512

LORIS-MRI Release v21.0.5

This release of the LORIS-MRI repository accompanies the LORIS 21.0.5 release.

This release fixes a bug introduced for phantom dataset where the candidate validation was done before checking if the scan was a phantom scan. The issue only happened when loading phantom scans via the insertion scripts.

LORIS MRI Release v22.0.0

This release of the LORIS-MRI repository accompanies the LORIS 22.0.0 release.

• List of all LORIS-MRI 22.0.0 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• Can now import MRI datasets already organized according the the BIDS standard (#499)

• Gzipping of the NIfTI files created by mnc2nii when createNifti option is set to Yes (#468)

Improvements

• Updated the Mac install README to more recent version of LORIS-MRI (#505)

• Removed the site question from the install script since this information is not used anywhere during the install process (#503)

• Updates to the documentation of LORIS-MRI (#502)

• Refactoring of how the config settings are fetched, for those of type string or boolean (#500 & #466)

• Removed the mri_acquisition_dates table and updated the queries associated to that table (#463)

• Exit the DICOM archive insertion if the insertion in the tarchive derived tables failed (#464)

• Added chunking of EDF files for future visualization of EEG datasets via the browser (#462)

• Simplification of the candidate validation code (#461)

• Fixed the query logic to select files for which NIfTI files should be created in mass_nii.pl (#458)

• Stops inserting a dump of the whole MINC header in the parameter_file table (#453)

• Bulletproof the way the host's IP address is computed in dicomTar.pl (#452)

• Fixed get_dicom_file.pl failure when the series description contained a sed special character (#434)

Notes for Existing Projects

• New dependencies (#499) were added to be able to create a pic of the NIfTI files imported via the BIDS structure. To install those dependencies, make sure to first source the LORIS-MRI python virtual environment (source /data/%PROJECT%/bin/mri/python_virtualenvs/loris-mri-python/bin/activate) and run the following commands (without sudo):

pip install matplotlib
pip install nose
pip install sklearn
pip install nilearn

• The following tool script can be run if you wish to gzip the NIfTI files already created by mnc2nii: tools/gzip_nifti_files.pl (#468)

• The profileTemplate.pl has been modified to remove the call to the function NeuroDB::MRI::subjectIDIsValid. This function has been removed from the MRI.pm library since that code was duplicated in MRIProcessingUtility.pm (#461). Ensure to remove the following from your profile file (typically called prod under dicom-archive/.loris-mri) if it was still there.

        # Note, this function will ensure that the PSCID/CandID information refers to a valid candidate
        # and it will check that the Visit label exists in the Visit_Windows table
        if(!NeuroDB::MRI::subjectIDIsValid($subjectID{'CandID'}, $subjectID{'PSCID'}, $subjectID{'visitLabel'}, $dbhr)) {
            return undef;
        }

• If you encounter the following error when running the perl imaging pipeline, update your DBI CPAN library to the latest version with cpan DBI (Reference).

Can't locate object method "selectall_array" via package "DBI::db" at /data/ccna/bin/mri/uploadNeuroDB/NeuroDB/MRIProcessingUtility.pm line 1819.

Clean up

• If you wish to delete all header entries (a.k.a. the full dump of the MINC header) from the parameter_file table, run the clean up patch called 2019_07_04_remove_header_row_from_parameter_file_and_convert_back_to_Value_field_to_text.sql

• If you wish to clean up and remove the mri_acquisition_dates table from your database, run the clean up patch called 2019-07-17_remove_mri_acquisition_dates_table.sql

LORIS MRI Release v21.0.0

This release of the LORIS-MRI repository accompanies the LORIS 21.0.0 release.

• List of all LORIS-MRI 21.0.0 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• Replaced Gender field with Sex in the candidate and tarchive tables (see #336, #382)

• New script to Delete uploaded scans (see #391 ), in the tools directory

• New script to generate BIDS-compliant structures for existing NiFTi files stored in your LORIS instance - tools/MakeNiiFilesBIDSCompliant.pl (see #198)

• Added the image_type header into the mri_protocol table for protocol discrimination of acquisitions. This header is particularly useful to differentiate phase and magnitude files of fieldmap acquisitions (the only difference between the two image types is in the image_type header) (see #416)

• Addition of a tool that produces 3D rendering of the face for visual QC of defaced images (see #415)

• Added a batch script to run defacing in parallel on multiple CPUs using QSUB if the Config Setting is_qsub is set (see #414)

• Grep the age of a candidate at the time of acquisition from the BIDS sub-XXX_scans.tsv file and insert it into physiological_parameter_file (see #447)

Improvements

• Cleaned up multiple database paths referring to the same path (see #399)

• Replaced CenterID field of the candidate table by RegistrationCenterID (see #341, #442)

• Refactoring of database calls using Moose (see #282, #295, #303, #311, #330)

• Replaced all hardcoded prod file mentions with (pre-existing) Config setting (see #254)

• Removed the unused column PendingStaging from the files table (see #292 and LORIS patch aces/Loris#3569)

• Updated the exit code of updateMRI_Upload.pl to use the ExitCode.pm (see #305)

• Harmonized the multiple ways of getting MINC header information in the code (see #351)

• Added missing CPAN dependencies in the install script of LORIS-MRI (see #439)

• Reordering of the environment file for new installs (see #148)

• Upgraded the BIDS insertion pipeline to use the more recent version of pybids (0.9.1) (see #448)

Notes for Existing Projects

Make sure to run the release patch on the LORIS side as there are significant SQL changes affecting the MRI side of LORIS.
Many core Loris-MRI scripts have now been updated with improved command-line arguments (see notes below.)

• Caveat for custom tools: replace Gender with Sex when querying the candidate or tarchive table

• Caveat for custom tools: replace CenterID with RegistrationCenterID when querying the candidate table

• Note Config path changes: Cleaned up multiple database paths referring to the same path (see #399)

• Make sure that your Config setting dataDirBasepath is configured to the proper data path (typically /data/%PROJECT%/data)

• Caveat for projects using the BIDS insertion pipeline. Since pybids changed the way files are queried in the most recent versions, the insertion LORIS-MRI BIDS insertion pipeline had to be refactored to work with the latest stable version of pybids (0.9.1). To upgrade your install of pybids, simply run in your python virtualenv (a.k.a. after having sourced your virtualenv: source $LORIS_MRI/python_virtualenvs/loris-mri-python):

pip uninstall pybids
pip install --upgrade pip
pip install pybids==0.9.1

• New CPAN dependencies to install:

sudo cpan install Moose
sudo cpan install MooseX::Privacy
sudo cpan install TryCatch
sudo cpan install Throwable

• Projects that plan on using the defacing scripts should reorder their environment files as shown in PR #429.

• The following scripts must now be run with the -profile <filename> option set (filename typically prod)

  • batch_upload_tarchive.pl
  • tools/seriesuid2fileid.pl

• Projects that call one of the following scripts using a cronjob should update their cronjob to add the .pl extension to the script:

  • batch_uploads_imageuploader => batch_uploads_imageuploader.pl
  • batch_uploads_tarchive => batch_uploads_tarchive.pl
  • tarchiveLoader => tarchiveLoader.pl
  • uploadNeuroDB/bin/Mincinfo_wrapper => uploadNeuroDB/bin/Mincinfo_wrapper.pl
  • uploadNeuroDB/bin/mincpik => uploadNeuroDB/bin/mincpik.pl
  • tools/seriesuid2fileid => tools/seriesuid2fileid.pl

• Before running the deletion script on an old upload (data uploaded before release 20.3), make sure that the script called cleanup_paths_of_violation_tables.pl (in the tools directory) from #403 (release 20.3) has been run to update all paths in mri_violations_log, mri_protocol_violated_scans and MRICandidateErrors to their actual path in the trashbin subdirectory of your LORIS-MRI data directory.

LORIS MRI Release v20.3.0

This release of the LORIS-MRI repository accompanies the LORIS 20.3.0 release.

• List of all LORIS-MRI 20.3 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• New tool run_defacing_script.pl to deface structural MINC images already registered in LORIS, and register the defaced images back into LORIS (#366 with SQL changes in aces/Loris#4232), using a modified version of the deface_minpipe.pl MINC tool that performs the defacing algorithm developed in the Collins lab at the MNI. (see #366)
• Images with no acquisition date (e.g. in the image header) can now be inserted. This is a new feature to facilitate Open Science projects where no dates should be available in the images' header. ( #395)

Improvements

• [Bug fix] Removed the option -clobber from the call to dicomTar.pl in ImagingUpload.pm so that DICOM archive cannot be overwritten by default by the insertion pipeline (see #396)

• [Bug fix] Use UploadID to update the mri_upload table instead of the TarchiveID when running the insertion pipeline (see #397)

• [Bug fix] Fix the paths stored in the violation tables (mri_violations_log, mri_violated_scans, MRICandidateErrors, see #403)

• [Bug fix] Added a check to verify if CandID and PSCID from the image's header are referring to the same candidate (see #407)

• [Bug fix] Circumvented bug that made tar fail on NFS file systems in get_dicom_script.pl and added an option -id (see #409)

• [Bug fix] Fixed the profileTemplate to include a call to the NeuroDB::MRI:: subjectIDIsValid function to validate subject ID information in getSubjectIDs of the profile file. (see #411)

Notes for Existing Projects

• Make sure to run the tool script called cleanup_paths_of_violation_tables.pl in order to update the MINC location in the 3 violation tables (mri_violations_log, mri_violated_scans, MRICandidateErrors, see #403)

• Because of the missing check (#407) between CandID and PSCID in the image header, it is possible that MINC files are not associated to the correct candidate. We are working on an MRI doctor script that will do all sort of data integrity checks and this will definitely be one of them. (see #400 for details)

• Recommended improvement: Validate your visit labels before loading scans: If the function getSubjectIDs in your prod file (i.e. the MRI configuration file in dicom-archive/.loris-mri/) contains the following flag $subjectID{'createVisitLabel'} set to 1 ($subjectID{'createVisitLabel'} = 1;), then we recommend adding a call to the NeuroDB::MRI:: subjectIDIsValid function to validate the visit label, otherwise the pipeline will automatically create the visit in the session table (see #411 for details).

• If you desire to use the new defacing tool, make sure you have:

  • downloaded and installed the bic-mni-models anatomical template library and the beast library from the MINC tools in /opt/minc/ (https://bic-mni.github.io/#data-files-and-models)
  • added the two following variables to your environment file so that the scripts can find the MNI models (replace %MINC_TOOLKIT_DIR% by the path to your MINC toolkit install):

  # for the defacing scripts
  export BEASTLIB=%MINC_TOOLKIT_DIR%/../share/beast-library-1.1
  export MNI_MODELS=%MINC_TOOLKIT_DIR%/../share/icbm152_model_09c
  

  • sourced the SQL patch from the aces/Loris#4232 LORIS PR

  • adapted the constant variables (%SPECIAL_ACQUISITIONS_FILTER, @MULTI_CONTRAST_ACQUISITIONS_BASE_NAMES) at the top of the tools/run_defacing_script.pl script to your project's need.

  • make sure which deface_minipipe.pl is returning the following $LORIS-MRI/uploadNeuroDB/bin/deface_minipipe.pl script and not the one present in the MINC toolkit (/opt/minc/$VERSION) as LORIS-MRI is using the version of deface_minipipe.pl under development (which have not yet been released with the MINC tools). If which deface_minipipe.pl is not pointing to the correct script, you will need to update the $PATH bash variable of your environment file to make sure $LORIS-MRI/uploadNeuroDB/bin is the first path mentioned in the variable.

LORIS MRI Release v20.2.0

This release of the LORIS-MRI repository accompanies the LORIS 20.2.0 release.

• List of all LORIS-MRI 20.2 changes (not including schema and module changes which were contributed to the main LORIS repo)

New Features

• EEG BIDS import pipeline scripts, written in python (#337, #347)

• Allow insertion of MINC files without associated DICOM files, or TarchiveID or UploadID values (#340): as of this release, it is possible to insert directly MINC files into the database without associated DICOM files using minc_insertion.pl directly.

• New generic (non-MINC) file image insertion script (#271): this script allows insertion of any non-MINC file image into the database via the script imaging_non_minc_insertion.pl to insert non-MINC files into the files table.

• New tool to select, extract and tar DICOM files for a specified regex matching the PatientName and/or acquisition protocol (#360)

• Added a check for the SeriesUID/echo_time pair before insertion into the database to avoid duplication of files in the database. If a file is already registered in the database with that same SeriesUID/echo_time pair of values, the insertion script will return a message saying the image is already inserted in the database with the name of the registered file (see #359 for more details)

Improvements

• Non DICOM files are now ignored when running the default imaging insertion (MRI) pipeline (see #371 and #373)

• Fixed a bug where a DICOM without the header field (0020,0032) was automatically excluded by get_dicom_info.pl because if was flagged as non-DICOM. With this update, running this script with the option -slicepos (#365) will include and load DICOMs that do not contain this header field.

• New Configuration module field under the Imaging pipeline section called "Modalities on which SNR should be calculated (compute_snr_modalities)". This replaces the getSNRModalities() function in the prod (profile configuration) file (#358)

• Fixed a bug when running the DTIPrep pipeline on datasets where NIfTI files are created at the time of insertion (before the bug fix, if the T1W file selected was the NIfTI file instead of the MINC file, then the MINC tools would fail running and the DTIPrep pipeline crashed; this has been fixed in #355).

• Fixed an error message that was displayed in the terminal when loading NeuroDB/MRI.pm library (#357)

• Uses perl function getpwuid instead of whoami fetch the Linux User ID (#368)

• Documentation for more perl scripts: dicomSummary.pl (#342) and get_dicom_info.pl (#343)

• Fixed cleanupTarchives.pl tool and documented it (#344)

• Fixed updateHeaders.pl tool and documented it (#345)

Notes for Existing Projects

• Important: Update your Configuration module, Imaging section -- New field: "Modalities on which SNR should be calculated (compute_snr_modalities)" AND remove from your prod file the function called getSNRModalities() (see #358)

• To use the EEG BIDS pipeline, simply follow the install instructions in #337, #347. For detailed information, consult the updated README.md and 02-Install.md under docs/

• To use the new generic (non-MINC) file image insertion script (#271), you will need to install the JSON CPAN library (may require sudo) :

cpan install JSON

LORIS MRI Release v20.1.0

This release of the LORIS-MRI repo accompanies the LORIS 20.1.0 release.

• List of all LORIS-MRI 20.1 changes (not including schema and module changes in the main LORIS repo)

New Features

• Added phantom validation to the imaging insertion pipeline, matching the DICOM header PatientName field against the appropriate phantom regex stored in the Config module (LegoPhantomRegex / LivingPhantomRegex) (see #338)

Improvements

• Creation of NIfTI .bvec and .bval files when converting MINC files to NIfTI for DWI acquisitions (see #335)

• Moved filterParameters() function out of the profileTemplate (prod) file since this function has never been customized by any project to our knowledge (see #309)

• Cleanup:

• Removal of deprecated tools from the LORIS-MRI repository (see #332)
• Removal of the creation of site-specific incoming directories from the install script since those folders are never used (see #334)
• Moved some scripts from dicom-archive to the tools directory (see #333)

Notes for Existing Projects

• Projects that have the option create_nii set to Yes in the Config module can mass-create all the missing NIfTI .bvec and .bval files for DWI acquisitions by running the create_nifti_bval_bvec.pl script present in the tools directory (see "Notes for existing projects" described in PR #335)

• Make sure to update the $PERL5LIB variable to add the path to your dicom-archive directory so that scripts moved to the tools directory can load perl libraries from dicom-archive/DICOM (see #333)

export PERL5LIB=/data/%PROJECT%/bin/mri/uploadNeuroDB:/data/%PROJECT%/bin/mri/dicom-archive:$PERL5LIB

• Existing projects can remove the filterParameter() function from your LORIS-MRI prod config file since it is not used anymore (see #309)

• For projects currently using site-specific incoming data directories, PR #334 should not impact this practice