Integration test to check that the ORM definitions match the LORIS SQL database #1198

maximemulder · 2024-10-07T17:17:59Z

Description of the PR

This PR adds an integration test that checks that the ORM definitions correspond to the SQL database as defined in the main LORIS repository.

It notably checks the following:

Check that each ORM model has a corresponding table.
Check that each ORM model has the same columns as its table.
Check that each ORM column has a similar type as its SQL column (on an abstract level, it does not check for details such as column length like INT(11) vs INTEGER, it does check for nullability).

To make this test possible, this PR also improves the LORIS-MRI Docker / integration setup, in particular the connection between the database image and the main LORIS-MRI image.

Notably, the Raisinbread database is now built directly into the database image, which allows to cache it with the image and not resource the SQL files each time the database image is ran (which is the default MariaDB Docker behaviour). This should nullify the need for LORIS-MRI Raisinbread overwrites, which could be removed in a follow-up PR. Note however that since both the main LORIS-MRI and database Docker images are quite large, it still takes ~3min to import the cached Docker layers.

There are also a few changes made to the models in this PR. Some are things caught by this PR (whooopsie !), some others are just improvements that this PR makes me confident enough to do.

cmadjar

Looks good. One tiny question to answer though.

cmadjar · 2024-10-16T17:58:33Z

python/tests/integration/test_orm_sql_sync.py

+            assert orm_column_python_type == sql_column_python_type
+            assert orm_column.nullable == sql_column.nullable
+
+        print()


any reason there is an empty print here? Maybe cosmetic?

@maximemulder maybe I should have tagged you so you could see the review?

Oh, it's just a statement to add an empty line break between each table in the output, the result looks like this:

test table: candidate test column: CandID test column: PSCID ... test column: Active test table: session test column: ID ...

Note that the output only appears when the test fails though. I should probably have added a fail commit in the history as a demonstration, I'll do that tomorrow morning (now is not really the time for that 😅).

@cmadjar See the "test error" commit integration test to see what the error looks like if the ORM mappings are wrong.

This reverts commit d1a70e6.

cmadjar

👍

* remove trailing whitespaces (#1149) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * SQLAlchemy proof-of-concept (#1152) * add sqlalchemy * fix lints * add comment --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * fix (#1168) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * integration test (#1138) * test * main Loris here * ignore Loris python * clone Loris * rm loris * override test * override test * override RB data * fix folder * test php files * test * try * test * try install loris mri * os * test * rm modules * Update flake8_python_linter.yml * Small linting improvements (#1167) * better linting * remove tab in comment --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * Improve Python subject determination and validation error handling (#1146) * improve subjects determination and validation * use tuple for database query arguments * query on single line --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * Add literal regex leading r (#1176) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * fix (#1178) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * LORIS-MRI tooling configuration (#1170) * add pyproject.toml * keep flake8 (for now) * double type checking * Use Pyright as main type checker * remove comment * test * test * change configuration * add python environment variables * try factorization * test * test * test * test * remove temporary things * test ruff error * ruff output github * further test errors * test pyright output github * test * test * test * change comment * remove test errors * test --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * update docker install (#1171) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * add RB override tables for the MRI side (#1173) * Remove php 8.1 and 8.2 from the LORIS tests to keep only test on PHP 8.3 (#1175) * remove php 8.1 and 8.2 testing * update to 8.3 in Dockerfile.test.php8 * ADD minc tools install in gitactions (#1179) * todo install loris mri * test * run pytest in gitaction (#1172) * test * ttt * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * Delete test_example.py * rename * done * Delete test/Dockerfile.test.py * test (#1187) * unit tests poc (#1188) * add more pytest global rules (#1194) * Stricter linter configuration (#1193) * stricter ruff config * do not ignore N818 * lint for sorted imports * add python scripts directory (#1196) * Bunch of SQLAlchemy definitions (#1192) * bunch of sqlalchemy definitions * add project orm definition * fix candidate to site query function * optimize docker image (#1195) * fix pytest config (#1197) * fix python scripts (#1199) * remove mysql-connector (#1200) * Improve subject configuration structure (#1150) * refacor subject config * remove suspicious phantom code * change naming * fix rebase lints * cleaner database url * Cecile CandID path join bug fix Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> --------- Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Integration test to check that the ORM definitions match the LORIS SQL database (#1198) * orm sync integration test * add docker health check * use integration config file * test error * Revert "test error" This reverts commit d1a70e6. * Clean up Docker database scripts (#1202) * clean docker database * remove docker warning * remove unused attribute (#1201) * try fix perl ci * Add file, file_parameter and scanner SQLAlchemy models (#1207) * add tables * rename file model * rename parameter_file * fix lints (#1210) * Unify LORIS-MRI logging (#1191) * new env * change print order in database log * create notification type if needed * fix typing pyright update * Integration test for `run_dicom_archive_loader.py` (#1203) add integration test put keys in github repo debugging add database check add comments and debugging remove s3 keys change converter in test check file tree fix doc fix docs 2 rename variable go back to using s3 keys fix path try fix * rename database sub-namepaces (#1214) * fix warnings (#1215) * Factorize get subject session (#1190) * factorize subject session * re-add site log * Migrate "get all sites from the database" to the SQLAlchemy abstraction (#1216) * Fix new PEP585 Ruff lint (#1218) * Clean-up database scan type (#1139) Complementary PR of LORIS #9304 * Migrate DICOM archive files to new database abstraction (#1217) * wip * use self.dicom_archive.id instead of self.tarchive_id * fix todo * Fix SQLAlchemy file model (#1221) * Improve Python requirements order (#1222) * add python documentation (#1223) * Update SQLAlchemy bindings to match LORIS#9556 (#1229) * trigger_tests * try update * fix old queries for integration tests * Migrate `ImagingUpload` and `MriUploadDB` to the new database abstraction (#1224) * migrate mri upload to sqlalchemy * fix concurrency bug * fix clean up * display mri upload ids use dicom archive id instead of tarchiveid because it is prettier ruff check * migrate scanner to new abstraction (#1232) * Fix install script (#1234) * fix install script and move files related to install into the install directory * fix install script and move files related to install into the install directory * fix install script and move files related to install into the install directory * fix paths in imaging_install_test.sh * fix paths in imaging_install_test.sh * fix SQL error in install script * fix tests * fix tests * fix github action * add cpanfile * copy cpanfile in MRI dockerfile * add new dependency in order to be able to source the Digest::BLAKE2 in the cpanfile * fix Digest::BLAKE2 installation though cpanfile * move config templates to a templates directory and requirements files into a requirements directory * remove copy * fix test * fix test * fix test * fix test * Maxime's feedback * specify python version (#1237) * fix large file hash memory (#1236) * add lib.util module (#1238) * New DICOM study import script (#1117) * rewrite dicom archive * update * remove --year and --target options * get session from config file * remove unused upload argument * fix little oopsie for --session * handle large files hash * fix unhandled modalities * use new lib.util module * Add support to be able to read headers from enhanced DICOMs (#7) * print * print * print * print * add support for enhanced DICOMs * add support for enhanced DICOMs * rework associate files with series * sort dicom series and files before inserting in the database * handle null scanner --------- Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * refactoring of CandID on the non-ORM python side * tests * fix insertion in mri_protocol_violation_scans * alphabetical order is hard sometimes * FIx `get_candidate_id` documentation --------- Co-authored-by: Maxime Mulder <maxime-mulder@outlook.com> Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> Co-authored-by: Shen <kongtiaowangshen@gmail.com>

maximemulder force-pushed the 2024-10-07_sqlalchemy-sync-integration-test branch 7 times, most recently from c6f1e70 to 561d19e Compare October 7, 2024 19:47

maximemulder changed the title ~~Database integration test~~ Integration test to check that ORM definitions match the LORIS SQL database Oct 7, 2024

maximemulder changed the title ~~Integration test to check that ORM definitions match the LORIS SQL database~~ Integration test to check that the ORM definitions match the LORIS SQL database Oct 7, 2024

maximemulder added this to the 27.0 milestone Oct 7, 2024

maximemulder force-pushed the 2024-10-07_sqlalchemy-sync-integration-test branch 7 times, most recently from 84e8b6a to de17fbf Compare October 7, 2024 22:14

orm sync integration test

5f45df0

maximemulder force-pushed the 2024-10-07_sqlalchemy-sync-integration-test branch 9 times, most recently from 1b2a50e to 0cd39df Compare October 14, 2024 20:54

maximemulder force-pushed the 2024-10-07_sqlalchemy-sync-integration-test branch 15 times, most recently from 6328cc4 to 2ec9013 Compare October 14, 2024 23:13

add docker health check

638406d

maximemulder force-pushed the 2024-10-07_sqlalchemy-sync-integration-test branch 3 times, most recently from ce7e375 to 7cf564a Compare October 15, 2024 00:58

use integration config file

340dc59

maximemulder force-pushed the 2024-10-07_sqlalchemy-sync-integration-test branch from 7cf564a to 340dc59 Compare October 15, 2024 01:03

maximemulder marked this pull request as ready for review October 15, 2024 01:04

maximemulder requested a review from cmadjar October 15, 2024 14:52

cmadjar reviewed Oct 16, 2024

View reviewed changes

maximemulder added 2 commits October 17, 2024 11:36

test error

d1a70e6

Revert "test error"

4cf3235

This reverts commit d1a70e6.

cmadjar approved these changes Oct 17, 2024

View reviewed changes

cmadjar merged commit c138f7d into aces:main Oct 17, 2024
9 checks passed

cmadjar added this to the 27.0 milestone Apr 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Integration test to check that the ORM definitions match the LORIS SQL database #1198

Integration test to check that the ORM definitions match the LORIS SQL database #1198

Uh oh!

maximemulder commented Oct 7, 2024 •

edited

Loading

Uh oh!

cmadjar left a comment

Uh oh!

cmadjar Oct 16, 2024

Uh oh!

cmadjar Oct 16, 2024

Uh oh!

maximemulder Oct 16, 2024 •

edited

Loading

Uh oh!

maximemulder Oct 17, 2024

Uh oh!

cmadjar left a comment

Uh oh!

Uh oh!

Uh oh!

Integration test to check that the ORM definitions match the LORIS SQL database #1198

Integration test to check that the ORM definitions match the LORIS SQL database #1198

Uh oh!

Conversation

maximemulder commented Oct 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the PR

Uh oh!

cmadjar left a comment

Choose a reason for hiding this comment

Uh oh!

cmadjar Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

cmadjar Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

maximemulder Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maximemulder Oct 17, 2024

Choose a reason for hiding this comment

Uh oh!

cmadjar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maximemulder commented Oct 7, 2024 •

edited

Loading

maximemulder Oct 16, 2024 •

edited

Loading