Improve subject configuration structure #1150

maximemulder · 2024-08-14T16:09:41Z

This PR makes the Python configuration file (database, S3, and subject IDs) more explicit by wrapping it in typed dataclasses instead of dicts, and adds a few comments.

I discussed some suspicious phantom-related code with @cmadjar, and it appears that phantom scans are not deeply tested in LORIS-MRI Python so there are likely a few bugs in our code. Therefore, I removed the suspicious lines as we probably want the scanner/scanner candidate to be created as early as possible so that the rest of the code does not have to worry about it.

Phantom configuration is still possible provided that the scanner and candidate exist in the database, which the user can add themselves since there is access to the database in the configuration function. More work is probably needed on that front (small quality-of-life improvements or the infamous phantom refactor) but that is a subject for other PRs.

NOTE FOR EXISTING PROJECTS: To update your project, change your configuration file (usually dicom-archive/.loris_mri/database_config.py) from the dict-shaped template to the typed one by imitating dicom_archive/database_config_template.py.

cmadjar · 2024-08-14T17:04:04Z

@maximemulder for phantoms, the PSCID is always scanner and looked up in the DB. Phantoms handling is horrible 🤣

cmadjar · 2024-08-14T17:09:09Z

@maximemulder can you give an example on how that would work with changes made to the template so I can get a better idea of how that would work.

Link to the database_config.py template: https://github.com/aces/Loris-MRI/blob/main/dicom-archive/database_config_template.py

maximemulder · 2024-08-14T17:41:19Z

@cmadjar The idea would be to return objects of the *Config classes defined in lib.dataclass instead of a dictionary, so instead of this:

def get_subject_ids(db, dicom_value=None, scanner_id=None):
    subject_id_dict = {}
    imaging = Imaging(db, False)

    phantom_match   = re.search('(pha)|(test)', dicom_value, re.IGNORECASE)
    candidate_match = re.search('([^_]+)_(\d+)_([^_]+)', dicom_value, re.IGNORECASE)

    if phantom_match:
        subject_id_dict['isPhantom']  = True
        subject_id_dict['CandID']     = imaging.get_scanner_candid(scanner_id)
        subject_id_dict['visitLabel'] = dicom_value.strip()
        subject_id_dict['createVisitLabel'] = 1
        subject_id_dict['ProjectID'] = 1
        subject_id_dict['CohortID'] = 1
    elif candidate_match:
        subject_id_dict['isPhantom']  = False
        subject_id_dict['PSCID']      = candidate_match.group(1)
        subject_id_dict['CandID']     = candidate_match.group(2)
        subject_id_dict['visitLabel'] = candidate_match.group(3)
        subject_id_dict['createVisitLabel'] = 0

    return subject_id_dict

We would would have this:

from lib.dataclass.config import SubjectConfig, CreateVisitConfig


def get_subject_ids(db: Database, subject_name: str, scanner_id: int | None = None) -> SubjectConfig | None:
    imaging = Imaging(db, False)

    phantom_match   = re.search('(pha)|(test)', subject_name, re.IGNORECASE)
    candidate_match = re.search('([^_]+)_(\d+)_([^_]+)', subject_name, re.IGNORECASE)

    if phantom_match:
        return SubjectConfig(
            name        = subject_name,
            is_phantom  = True,
            psc_id      = 'scanner',
            cand_id     = imaging.get_scanner_candid(scanner_id),
            visit_label = subject_name.strip()
            create_visit = CreateVisitConfig(
               project_id = 1,
               cohort_id  = 1,
            )
        )
    elif candidate_match:
        return SubjectConfig(
            name        = subject_name,
            is_phantom  = False,
            psc_id      = candidate_match.group(1),
            cand_id     = int(candidate_match.group(2)),
            visit_label = candidate_match.group(3),
            create_visit = None,
        )

    return None

I note that the current function has a parameter dicom_value = None, I think subject_name would be a better name (there are many values in a DICOM, and this name come from a file name as well), and I don't think this should ever be None no ?

This change can also be done with other currently non-implemented *Config classes like DatabaseConfig or ApiConfig.

maximemulder

I'd like your opinion on these two conditions @cmadjar . I am very confident on the issues of these conditions, but would just like to know whether this may be an indicator of a larger problem.

python/lib/dcm2bids_imaging_pipeline_lib/base_pipeline.py

maximemulder · 2024-09-25T20:03:50Z

I edited the PR description with up-to-date information. Manually tested with run_dicom_archive_loader.py and it worked correctly. I remind that this is a breaking change for every project and dev as it requires to redo the Python configuration. However this new configuration should be easier and less error-prone.

Ready for review or merge @cmadjar (or re-assign someone else to review if you don't have time).

EDIT: Just fixed a typo in a comment but still ready to merge.

maximemulder · 2024-10-01T22:30:47Z

Naming changes:

config.py ===> config_file.py
subject / x_subject_ids / SubjectConfig ===> subject_info / x_subject_info / SubjectInfo

The config file change is to not conflict with the config from the database.
The subject change is to have a more uniform terminology, and is also more representative of the data and less ambiguous since it represents various information about a subject, not only IDs in the database.

cmadjar · 2024-10-04T15:17:05Z

@maximemulder As discussed, if you could add a note in the description for existing project on how to migrate their configuration file.

I will work on upgrading my HBCD sandbox to 26 and should be able to test the PR next week 🤞

maximemulder · 2024-10-04T16:13:58Z

@maximemulder As discussed, if you could add a note in the description for existing project on how to migrate their configuration file.

Done !

cmadjar

@maximemulder I am not able to connect to MySQL with the new way of connecting. I get the following error:

$ run_dicom_archive_validation.py -p database_config_27.py -u 140 -t /data/loris/data/tarchive/DCM_2023-11-28_DICOM_DIR_2024-10-09_18h28m05s_d8kht8nu.tar 

Traceback (most recent call last):
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/network.py", line 482, in open_connection
    addrinfos = socket.getaddrinfo(self.server_host,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
          ^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
    with util.safe_reraise():
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
    return self._create_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
    return _ConnectionRecord(self)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
    self.__connect()
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
    with util.safe_reraise():
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/__init__.py", line 179, in connect
    return MySQLConnection(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/connection.py", line 95, in __init__
    self.connect(**kwargs)
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/abstracts.py", line 716, in connect
    self._open_connection()
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/connection.py", line 206, in _open_connection
    self._socket.open_connection()
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/network.py", line 500, in open_connection
    raise errors.InterfaceError(
mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'd@localhost:3306' (-2 Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/loris/bin/mri/python/scripts/run_dicom_archive_validation.py", line 75, in <module>
    main()
  File "/opt/loris/bin/mri/python/scripts/run_dicom_archive_validation.py", line 71, in main
    DicomValidationPipeline(loris_getopt_obj, os.path.basename(__file__[:-3]))
  File "/opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/dicom_validation_pipeline.py", line 30, in __init__
    self.validate_subject_info()
  File "/opt/loris/bin/mri/python/lib/dcm2bids_imaging_pipeline_lib/base_pipeline.py", line 241, in validate_subject_info
    validate_subject_info(self.db_orm, self.subject_info)
  File "/opt/loris/bin/mri/python/lib/validate_subject_info.py", line 17, in validate_subject_info
    candidate = try_get_candidate_with_cand_id(db, subject_info.cand_id)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/loris/bin/mri/python/lib/db/query/candidate.py", line 14, in try_get_candidate_with_cand_id
    return db.execute(query).scalar_one_or_none()
           ^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
    conn = self._connection_for_bind(bind)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in _connection_for_bind
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
    conn = bind.connect()
           ^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 148, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2442, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 449, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
    rec = pool._do_get()
          ^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
    with util.safe_reraise():
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
    return self._create_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
    return _ConnectionRecord(self)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
    self.__connect()
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
    with util.safe_reraise():
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 643, in connect
    return dialect.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 621, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/__init__.py", line 179, in connect
    return MySQLConnection(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/connection.py", line 95, in __init__
    self.connect(**kwargs)
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/abstracts.py", line 716, in connect
    self._open_connection()
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/connection.py", line 206, in _open_connection
    self._socket.open_connection()
  File "/home/lorisadmin/miniconda3/envs/loris-mri-py311/lib/python3.11/site-packages/mysql/connector/network.py", line 500, in open_connection
    raise errors.InterfaceError(
sqlalchemy.exc.InterfaceError: (mysql.connector.errors.InterfaceError) 2003: Can't connect to MySQL server on 'd@localhost:3306' (-2 Name or service not known)
(Background on this error at: https://sqlalche.me/e/20/rvf5)

I tried connecting manually to mysql with mysql -P 3306 -u lorisuser -h localhost -p and the correct password. It worked so this is not due to an incorrect setup.

mysql: DatabaseConfig = DatabaseConfig (
    host='localhost',
    username='lorisuser',
    password='################',
    database='prod_26',
    port=3306
)

Any idea what might be going on?

maximemulder · 2024-10-10T13:36:48Z

@cmadjar Hhhhm, no idea what's happening here since it is working well on my end. The last line seems to be implying that it is using a wrong username (d@localhost:3306) ? I guess you could print the URL in lib.db.connect.connect_to_database to be sure. If that's not it maybe another dependency problem ?

cmadjar

@maximemulder see one tiny bug on the push to s3 script

python/lib/dcm2bids_imaging_pipeline_lib/push_imaging_files_to_s3_pipeline.py

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

cmadjar

All good now! Thanks!

* remove trailing whitespaces (#1149) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * SQLAlchemy proof-of-concept (#1152) * add sqlalchemy * fix lints * add comment --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * fix (#1168) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * integration test (#1138) * test * main Loris here * ignore Loris python * clone Loris * rm loris * override test * override test * override RB data * fix folder * test php files * test * try * test * try install loris mri * os * test * rm modules * Update flake8_python_linter.yml * Small linting improvements (#1167) * better linting * remove tab in comment --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * Improve Python subject determination and validation error handling (#1146) * improve subjects determination and validation * use tuple for database query arguments * query on single line --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * Add literal regex leading r (#1176) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * fix (#1178) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * LORIS-MRI tooling configuration (#1170) * add pyproject.toml * keep flake8 (for now) * double type checking * Use Pyright as main type checker * remove comment * test * test * change configuration * add python environment variables * try factorization * test * test * test * test * remove temporary things * test ruff error * ruff output github * further test errors * test pyright output github * test * test * test * change comment * remove test errors * test --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * update docker install (#1171) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * add RB override tables for the MRI side (#1173) * Remove php 8.1 and 8.2 from the LORIS tests to keep only test on PHP 8.3 (#1175) * remove php 8.1 and 8.2 testing * update to 8.3 in Dockerfile.test.php8 * ADD minc tools install in gitactions (#1179) * todo install loris mri * test * run pytest in gitaction (#1172) * test * ttt * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * Delete test_example.py * rename * done * Delete test/Dockerfile.test.py * test (#1187) * unit tests poc (#1188) * add more pytest global rules (#1194) * Stricter linter configuration (#1193) * stricter ruff config * do not ignore N818 * lint for sorted imports * add python scripts directory (#1196) * Bunch of SQLAlchemy definitions (#1192) * bunch of sqlalchemy definitions * add project orm definition * fix candidate to site query function * optimize docker image (#1195) * fix pytest config (#1197) * fix python scripts (#1199) * remove mysql-connector (#1200) * Improve subject configuration structure (#1150) * refacor subject config * remove suspicious phantom code * change naming * fix rebase lints * cleaner database url * Cecile CandID path join bug fix Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> --------- Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Integration test to check that the ORM definitions match the LORIS SQL database (#1198) * orm sync integration test * add docker health check * use integration config file * test error * Revert "test error" This reverts commit d1a70e6. * Clean up Docker database scripts (#1202) * clean docker database * remove docker warning * remove unused attribute (#1201) * try fix perl ci * Add file, file_parameter and scanner SQLAlchemy models (#1207) * add tables * rename file model * rename parameter_file * fix lints (#1210) * Unify LORIS-MRI logging (#1191) * new env * change print order in database log * create notification type if needed * fix typing pyright update * Integration test for `run_dicom_archive_loader.py` (#1203) add integration test put keys in github repo debugging add database check add comments and debugging remove s3 keys change converter in test check file tree fix doc fix docs 2 rename variable go back to using s3 keys fix path try fix * rename database sub-namepaces (#1214) * fix warnings (#1215) * Factorize get subject session (#1190) * factorize subject session * re-add site log * Migrate "get all sites from the database" to the SQLAlchemy abstraction (#1216) * Fix new PEP585 Ruff lint (#1218) * Clean-up database scan type (#1139) Complementary PR of LORIS #9304 * Migrate DICOM archive files to new database abstraction (#1217) * wip * use self.dicom_archive.id instead of self.tarchive_id * fix todo * Fix SQLAlchemy file model (#1221) * Improve Python requirements order (#1222) * add python documentation (#1223) * Update SQLAlchemy bindings to match LORIS#9556 (#1229) * trigger_tests * try update * fix old queries for integration tests * Migrate `ImagingUpload` and `MriUploadDB` to the new database abstraction (#1224) * migrate mri upload to sqlalchemy * fix concurrency bug * fix clean up * display mri upload ids use dicom archive id instead of tarchiveid because it is prettier ruff check * migrate scanner to new abstraction (#1232) * Fix install script (#1234) * fix install script and move files related to install into the install directory * fix install script and move files related to install into the install directory * fix install script and move files related to install into the install directory * fix paths in imaging_install_test.sh * fix paths in imaging_install_test.sh * fix SQL error in install script * fix tests * fix tests * fix github action * add cpanfile * copy cpanfile in MRI dockerfile * add new dependency in order to be able to source the Digest::BLAKE2 in the cpanfile * fix Digest::BLAKE2 installation though cpanfile * move config templates to a templates directory and requirements files into a requirements directory * remove copy * fix test * fix test * fix test * fix test * Maxime's feedback * specify python version (#1237) * fix large file hash memory (#1236) * add lib.util module (#1238) * New DICOM study import script (#1117) * rewrite dicom archive * update * remove --year and --target options * get session from config file * remove unused upload argument * fix little oopsie for --session * handle large files hash * fix unhandled modalities * use new lib.util module * Add support to be able to read headers from enhanced DICOMs (#7) * print * print * print * print * add support for enhanced DICOMs * add support for enhanced DICOMs * rework associate files with series * sort dicom series and files before inserting in the database * handle null scanner --------- Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * refactoring of CandID on the non-ORM python side * tests * fix insertion in mri_protocol_violation_scans * alphabetical order is hard sometimes * FIx `get_candidate_id` documentation --------- Co-authored-by: Maxime Mulder <maxime-mulder@outlook.com> Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> Co-authored-by: Shen <kongtiaowangshen@gmail.com>

maximemulder mentioned this pull request Aug 14, 2024

Improve Python subject determination and validation error handling #1146

Merged

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch from 4bb6c5b to 4a4e23f Compare August 23, 2024 21:13

maximemulder mentioned this pull request Aug 23, 2024

SQLAlchemy proof-of-concept #1152

Merged

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch 3 times, most recently from 74e7698 to f77d9a7 Compare August 26, 2024 14:05

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch 5 times, most recently from 508d6f8 to c3d8a19 Compare September 18, 2024 18:25

maximemulder marked this pull request as ready for review September 18, 2024 18:26

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch 4 times, most recently from b2130a8 to 4cd7f09 Compare September 25, 2024 14:26

maximemulder commented Sep 25, 2024

View reviewed changes

python/lib/dcm2bids_imaging_pipeline_lib/base_pipeline.py Outdated Show resolved Hide resolved

python/lib/dcm2bids_imaging_pipeline_lib/base_pipeline.py Outdated Show resolved Hide resolved

maximemulder added the Caveat for Existing Projects label Sep 25, 2024

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch from cf0c600 to 72a546f Compare September 25, 2024 20:11

maximemulder mentioned this pull request Sep 27, 2024

Factorize get subject session #1190

Merged

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch from 9f144dd to 23bc589 Compare October 1, 2024 22:28

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch from 23bc589 to 9b003d5 Compare October 2, 2024 06:08

cmadjar assigned maximemulder Oct 4, 2024

cmadjar added this to the 27.0 milestone Oct 4, 2024

cmadjar added the Refactoring label Oct 4, 2024

maximemulder added the Difficulty: Medium PR or issue that require a moderate effort or expertise to implement, review, or test label Oct 4, 2024

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch from 9b003d5 to 7394197 Compare October 4, 2024 19:34

maximemulder mentioned this pull request Oct 7, 2024

Integration test to check that the ORM definitions match the LORIS SQL database #1198

Merged

cmadjar reviewed Oct 9, 2024

View reviewed changes

maximemulder mentioned this pull request Oct 10, 2024

Remove mysql-connector dependency #1200

Merged

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch from 7394197 to ad87522 Compare October 10, 2024 19:05

maximemulder added 4 commits October 10, 2024 15:06

refacor subject config

cb2200a

remove suspicious phantom code

2699ce8

change naming

95c2daa

fix rebase lints

50ff869

maximemulder force-pushed the 2024-08-08_refactor-subject-dict branch from ad87522 to 50ff869 Compare October 10, 2024 19:07

cmadjar requested changes Oct 10, 2024

View reviewed changes

python/lib/dcm2bids_imaging_pipeline_lib/push_imaging_files_to_s3_pipeline.py Outdated Show resolved Hide resolved

maximemulder and others added 2 commits October 10, 2024 15:27

cleaner database url

5ff68bf

Cecile CandID path join bug fix

84c5103

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

cmadjar approved these changes Oct 10, 2024

View reviewed changes

cmadjar merged commit 06c08e6 into aces:main Oct 10, 2024
9 checks passed

cmadjar added this to the 27.0 milestone Apr 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve subject configuration structure #1150

Improve subject configuration structure #1150

Uh oh!

maximemulder commented Aug 14, 2024 •

edited

Loading

Uh oh!

cmadjar commented Aug 14, 2024

Uh oh!

cmadjar commented Aug 14, 2024

Uh oh!

maximemulder commented Aug 14, 2024 •

edited

Loading

Uh oh!

maximemulder left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

maximemulder commented Sep 25, 2024 •

edited

Loading

Uh oh!

maximemulder commented Oct 1, 2024 •

edited

Loading

Uh oh!

cmadjar commented Oct 4, 2024

Uh oh!

maximemulder commented Oct 4, 2024 •

edited

Loading

Uh oh!

cmadjar left a comment

Uh oh!

maximemulder commented Oct 10, 2024

Uh oh!

cmadjar left a comment

Uh oh!

Uh oh!

cmadjar left a comment

Uh oh!

Uh oh!

Uh oh!

Improve subject configuration structure #1150

Improve subject configuration structure #1150

Uh oh!

Conversation

maximemulder commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmadjar commented Aug 14, 2024

Uh oh!

cmadjar commented Aug 14, 2024

Uh oh!

maximemulder commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maximemulder left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maximemulder commented Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maximemulder commented Oct 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmadjar commented Oct 4, 2024

Uh oh!

maximemulder commented Oct 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmadjar left a comment

Choose a reason for hiding this comment

Uh oh!

maximemulder commented Oct 10, 2024

Uh oh!

cmadjar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cmadjar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maximemulder commented Aug 14, 2024 •

edited

Loading

maximemulder commented Aug 14, 2024 •

edited

Loading

maximemulder left a comment •

edited

Loading

maximemulder commented Sep 25, 2024 •

edited

Loading

maximemulder commented Oct 1, 2024 •

edited

Loading

maximemulder commented Oct 4, 2024 •

edited

Loading