SQLAlchemy proof-of-concept

.github/workflows/flake8_python_linter.yml

@@ -7,36 +7,32 @@ name: Flake8 Linter
 on: pull_request
 
 jobs:
-  
+
   flake8-lint:
     runs-on: ubuntu-latest
-    
+
     steps:
       - name: Check out source repository
         uses: actions/checkout@v2
-      
+
       - name: Set up Python environment
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
-          python-version: "3.8"
-      
+          python-version: "3.11"
+
       - name: Flake8 Lint
         uses: reviewdog/action-flake8@v3
         with:
           # For more flake 8 arguments, visit the link below:
           # https://flake8.pycqa.org/en/latest/user/options.html
           # Currently, flake8 ignores the following;
-          #     W291: Trailing whitespace
-          #     W292: No newline at end of file
-          #     W293: Blank line contains whitespace
           #     E202: Whitespace before ')'
           #     E203: Whitespace before ':'
           #     E221: Multiple spaces before operator
           #     E241: Multiple spaces after ','
           #     E251: Unexpected spaces around keyword / parameter equals
           #     E272: Multiple spaces before keyword
-          flake8_args: "--ignore=W291,W292,W293,W503,E202,E203,E221,E241,E251,E272,E126,E131,E121,E111,E114,E127
+          flake8_args: "--ignore=W503,E202,E203,E221,E241,E251,E272,E126,E131,E121,E111,E114,E127
                         --max-line-length 120
                         --exclude python/react-series-data-viewer/protocol_buffers/"
           github_token: ${{ secrets.GITHUB_TOKEN }}
-

README.md

@@ -3,22 +3,22 @@ This Readme covers release 26.0.* of the LORIS Imaging Insertion Pipeline for Ub
 This repo accompanies the [LORIS neuroimaging data platform main repo](https://github.com/aces/Loris/releases)</b>, release 24.1.*.<br>
 For documentation and detailed setup information, please see the [LORIS-MRI documentation](docs/) for your installed version</b>.
 
-This repo can be installed on the same VM as the main LORIS codebase, or on a different machine such as a designated fileserver where large imaging filesets are to be stored. 
+This repo can be installed on the same VM as the main LORIS codebase, or on a different machine such as a designated fileserver where large imaging filesets are to be stored.
 
 # System Requirements
 
 * Perl
-* Python 3 with pip3 and virtualenv (step 2 below)
+* Python 3.11 with pip3 and venv (step 2 below)
 * MINC toolkit (step 3 below)
 * DICOM toolkit (step 4 below)
 * tpcclib (for HRRT PET only) (step 5 below)
 
-On <u>Ubuntu</u>, DICOM toolkit will be installed by the imaging install script (step 4 below). This script will _apt-get install dcmtk_.   
+On <u>Ubuntu</u>, DICOM toolkit will be installed by the imaging install script (step 4 below). This script will _apt-get install dcmtk_.
 
 For <u>CentOS</u>: The [LORIS wiki](https://github.com/aces/Loris/wiki/Imaging-Database) Imaging setup page (see Section 1, installing codebase) includes links to older transcripts for [CentOS installation](https://github.com/aces/Loris/wiki/CentOS-Imaging-installation-transcript) and notes on dependencies including [DICOM toolkit](https://github.com/aces/Loris/wiki/CentOS-Imaging-installation-transcript#7-install-dicom-toolkit).
 
 The following installation should be run by the `$lorisadmin` user. `sudo` permission is required.
-See [aces/Loris](https://github.com/aces/loris) README.md for further information. 
+See [aces/Loris](https://github.com/aces/loris) README.md for further information.
 
 # Dependencies
 
@@ -54,21 +54,19 @@ sudo chown -R lorisadmin:lorisadmin /opt/$projectname
 cd /opt/$projectname/bin
 ```
 
-Get the code: Download the latest release from the 
-[releases page](https://github.com/aces/Loris-MRI/releases) 
+Get the code: Download the latest release from the
+[releases page](https://github.com/aces/Loris-MRI/releases)
 and extract it to `/opt/$projectname/bin/mri`
 
-#### 2. Install Python 3 with `pip` and `virtualenv`
+#### 2. Install Python 3.11 with `pip` and `venv`
 
 ```bash
-sudo apt install python3 
-sudo apt install python3-dev
+sudo apt install python3.11 python3.11-dev python3.11-venv
 sudo apt install python3-pip
 sudo apt install libmysqlclient-dev
-sudo apt install virtualenv
 ```
 
-#### 3. Install MINC toolkit from http://bic-mni.github.io/ 
+#### 3. Install MINC toolkit from http://bic-mni.github.io/
 
 - Install MINC dependencies:
 
@@ -82,7 +80,7 @@ sudo apt-get install libgl1-mesa-glx libglu1-mesa
 
 - Download the MINC pre-compiled package for your operating system from http://bic-mni.github.io/.
 
-- Then install the MINC toolkit package: 
+- Then install the MINC toolkit package:
 
 ```bash
 # main minc lib
@@ -112,31 +110,31 @@ sudo dpkg -i beast-library-<version>.deb
 
 #### 4. Run installer to set up directories, configure environment, install Perl libraries and DICOM toolkit:
 
-```bash 
+```bash
 cd /opt/$projectname/bin/mri/install/
 bash ./imaging_install.sh
 ```
 
-You will be asked for the following input: 
+You will be asked for the following input:
 
 * What is the database name? $dbname
 * What is the database host? $dbhost
 * What is the MySQL user? $lorisuser [Use the same mysql user from the Loris installation, i.e. _lorisuser_]
-* What is the MySQL password? 
+* What is the MySQL password?
 * What is the Linux user which the installation will be based on? $lorisadmin
 * What is the project name? $projectname
-* What is your email address? 
+* What is your email address?
 * What prod file name would you like to use? default: prod  [leave blank]
 
-If the imaging install script reports errors in creating directories 
-(due to `/data/` mount permissions), review and manually execute 
-`mkdir/chmod/chown` commands starting at 
+If the imaging install script reports errors in creating directories
+(due to `/data/` mount permissions), review and manually execute
+`mkdir/chmod/chown` commands starting at
 [imaging_install.sh:L97](https://github.com/aces/Loris-MRI/blob/main/install/imaging_install.sh#L97)
 
-Note: The installer will allow Apache to write to the `/data/` and `/opt/` directories by 
-adding user `lorisadmin` to the Apache linux group.  To ensure this change takes 
-effect, log out and log back into your terminal session before running the 
-imaging pipeline. The installer will also set Apache group ownership of certain 
+Note: The installer will allow Apache to write to the `/data/` and `/opt/` directories by
+adding user `lorisadmin` to the Apache linux group.  To ensure this change takes
+effect, log out and log back into your terminal session before running the
+imaging pipeline. The installer will also set Apache group ownership of certain
 `/data/` and `/opt/` subdirectories.
 
 #### 5. HRRT PET
@@ -157,7 +155,7 @@ Then source the `.bashrc` file.
 
 **INSTALLATION COMPLETE!**
 
-Please refer to the [Install](docs/02-Install.md) section in the 
+Please refer to the [Install](docs/02-Install.md) section in the
 [LORIS-MRI documentation](docs/) for your installed version for:
 - customizations and protocol configurations ([Section 2.2](docs/02-Install.md#configuration)).
 - verifying that certain fields were correctly populated by `imaging_install.sh`

install/imaging_install.sh

@@ -115,7 +115,7 @@ echo
 echo "Creating loris-mri Python virtualenv in $mridir/python_virtualenvs/loris-mri-python/"
 # create a directory in $mridir that will store python 3 virtualenv
 sudo -S su $USER -c "mkdir -m 770 -p $mridir/python_virtualenvs/loris-mri-python"
-virtualenv $mridir/python_virtualenvs/loris-mri-python -p `which python3`
+python3.11 -m venv $mridir/python_virtualenvs/loris-mri-python
 source $mridir/python_virtualenvs/loris-mri-python/bin/activate
 echo "Installing the Python libraries into the loris-mri virtualenv..."
 pip3 install -r "$mridir/python/requirements.txt"

python/bids_import.py

@@ -10,12 +10,12 @@
 import lib.exitcode
 import lib.utilities
 import lib.physiological
-from lib.database   import Database
-from lib.candidate  import Candidate
+from lib.database_lib import Database
+from lib.candidate import Candidate
 from lib.bidsreader import BidsReader
-from lib.session    import Session
-from lib.eeg        import Eeg
-from lib.mri        import Mri
+from lib.session import Session
+from lib.eeg import Eeg
+from lib.mri import Mri
 from lib.database_lib.config import Config
 
 __license__ = "GPLv3"

python/extract_eeg_bids_archive.py

@@ -7,7 +7,7 @@
 import re
 from lib.lorisgetopt import LorisGetOpt
 from lib.imaging_io import ImagingIO
-from lib.database import Database
+from lib.database_lib import Database
 from lib.database_lib.config import Config
 from lib.exitcode import SUCCESS, BAD_CONFIG_SETTING
 from lib.log import Log

python/ingest_eeg_bids_datasets.py

@@ -5,7 +5,7 @@
 import os
 import sys
 from lib.lorisgetopt import LorisGetOpt
-from lib.database import Database
+from lib.database_lib import Database
 from lib.database_lib.config import Config
 from lib.exitcode import SUCCESS, INVALID_ARG
 import subprocess
@@ -112,7 +112,7 @@ def main():
       if not session_data:
         print('Session ID ' + eeg_dataset['SessionID'] + ' associated with UploadID ' + uploadid + ' does not exist.')
         sys.exit(INVALID_ARG)
-         
+
       candid = session_data[0]['CandID']
       pscid = session_data[0]['PSCID']
       visit = session_data[0]['Visit_label']
@@ -150,13 +150,13 @@ def main():
       # Assume eeg and raw data for now
       eeg_path = os.path.join(path, 'eeg')
       command = 'python ' + script + ' -p ' + profile + ' -d ' + eeg_path + ' --nobidsvalidation --nocopy --type raw'
-      
+
       try:
         result = subprocess.run(command, shell = True, capture_output=True)
 
         if result.stdout:
           print(result.stdout.decode('utf-8'))
-        
+
         if result.stderr:
           print(
             f'ERROR: EEG Dataset with uploadID {uploadid} ingestion log:\n ' + result.stderr.decode('utf-8')
@@ -177,7 +177,7 @@ def main():
         "UPDATE electrophysiology_uploader SET Status = 'Failed Ingestion' WHERE UploadID = %s",
         (uploadid,)
       )
-      
+
     # TODO: reupload of archive after ingestion
     # Delete if already exist
 

python/lib/__init__.py

@@ -1,4 +1,3 @@
-
 """
 From the documentation at https://docs.python.org/2/tutorial/modules.html#packages
 The __init__.py files are required to make Python treat the directories as
@@ -7,4 +6,4 @@
 the module search path. In the simplest case, __init__.py can just be an empty
 file, but it can also execute initialization code for the package or set the
 __all__ variable, described later.
-"""
+"""

python/lib/aws_s3.py

@@ -82,9 +82,9 @@ def check_object_content_exists(self, file_path, key):
             etag = lib.utilities.compute_md5_hash(file_path)
             self.s3_client.head_object(Bucket=self.bucket_name, Key=key, IfMatch=etag)
         except ClientError:
-            """            
+            """
             Per Boto3 documentation for S3.Client.head_object IfMatch will:
-            Return the object only if its entity tag (ETag) is the same as the one specified; 
+            Return the object only if its entity tag (ETag) is the same as the one specified;
             otherwise, return a 412 (precondition failed) error.
             """
             return False

python/lib/bidsreader.py

@@ -97,8 +97,8 @@ def load_bids_data(self, validate):
         # BIDSLayoutIndexer is required for PyBIDS >= 0.12.1
         # bids_pack_version = list(map(int, bids.__version__.split('.')))
         # disabled until is a workaround for https://github.com/bids-standard/pybids/issues/760 is found
-        # [file] bids_import.py 
-        # [function] read_and_insert_bids 
+        # [file] bids_import.py
+        # [function] read_and_insert_bids
         # [line] for modality in row['modalities']: (row['modalities'] is empty)
         # if (bids_pack_version[0] > 0
         #    or bids_pack_version[1] > 12
@@ -147,7 +147,7 @@ def load_candidates_from_bids(self):
         else:
             bids_subjects = self.bids_layout.get_subjects()
             participants_info = [{'participant_id': sub_id} for sub_id in bids_subjects]
-        
+
         if self.verbose:
             print('\t=> List of participants found:')
             for participant in participants_info:

python/lib/candidate.py

@@ -16,7 +16,7 @@ class Candidate:
     :Example:
 
         from lib.candidate import Candidate
-        from lib.database  import Database
+        from lib.database_lib import Database
 
         # database connection
         db = Database(config.mysql, verbose)