integration test #1138

kongtiaowang · 2024-06-20T14:01:52Z

This PR will run the sql files in RB_SQL folder without running other integration tests from core.

.github/workflows/flake8_python_linter.yml

.github/workflows/loristest.yml

test/RB_SQL/RB_Config.sql

cmadjar · 2024-08-23T18:13:59Z

test/RB_SQL/RB_Config.sql

+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (19,23,'0');
+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (20,24,'Customize this text with your citation policy (via Configuration module)');
+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (22,12,'Ymd');
+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (23,27,'/data-raisinbread/');


Config 27 is there @laemtl

cmadjar · 2024-08-23T18:14:13Z

test/RB_SQL/RB_Config.sql

+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (28,32,'/data/');
+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (29,33,'tools/logs/');
+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (30,34,'/data/incoming/');
+INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (31,35,'/data/demo/bin/mri/');


and config 35 is here @laemtl

cmadjar · 2024-08-23T19:22:38Z

.github/workflows/loristest.yml

+        cp test/Dockerfile.test.php8 Loris/
+        cp test/docker-compose.yml Loris/
+        cp test/phpunit.xml Loris/test/
+        cp test/RB_SQL/* Loris/raisinbread/RB_files/


Could we select here the RB SQL tables we want to source? Technically, we only need the following tables:

Config

ConfigSettings

ImagingFileTypes

Project

Visit_Windows

all the bids-* tables

candidate

cohort

mri_protocol_* tables

mri_scan_type

parameter_type (however, this could be populated automatically when running the pipeline)

permissions* (I would guess this would be needed when using the API)

project_cohort_rel

psc

session

user_perm_rel

user_project_rel

user_psc_rel

users

visit

visit_project_cohort_rel

Purposely skipping the following tables:

feedback_mri_* (don't need feedbacks for LORIS-MRI)

files (would use a very lightweight version of this table)

files_intermediary

files_qcstatus (QC so not really used by LORIS-MRI)

electrophysiology_uploader (not sure if this is needed)

hed_schema (not sure if this has data or has set up information, would need @jeffersoncasimir to decide)

hed_schema_nodes (same, @jeffersoncasimir)

hrrt_* (empty tables in RB)

mri_parameter_form ( I don't think there is any use of that table for LORIS-MRI)

mri_processing_protocol (empty table in RB)

mri_scanner (has scanner data but the pipeline is supposed to create that if it does not exist so maybe we can ensure the logic of creating the scanner works if we skip this table?)

mri_upload* (could have a lightweight version of it or keep it empty)

mri_violations_log (could have a lightweight version of it or keep it empty)

notification_spool (we don't really need all of that data)

parameter_files (could have a lightweight version of it or keep it empty)

physiological* tables (could have a lightweight version of it or keep it empty and the script will populate those)

server_processes (could probably keep that empty)

tarchive* (would have a lightweight version of it with a couple of files)

violations_resolved (could have a lightweight version of it or keep it empty)

yes, you can do it override any files under the RB_SQL folder.

* remove trailing whitespaces (#1149) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * SQLAlchemy proof-of-concept (#1152) * add sqlalchemy * fix lints * add comment --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * fix (#1168) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * integration test (#1138) * test * main Loris here * ignore Loris python * clone Loris * rm loris * override test * override test * override RB data * fix folder * test php files * test * try * test * try install loris mri * os * test * rm modules * Update flake8_python_linter.yml * Small linting improvements (#1167) * better linting * remove tab in comment --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * Improve Python subject determination and validation error handling (#1146) * improve subjects determination and validation * use tuple for database query arguments * query on single line --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * Add literal regex leading r (#1176) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * fix (#1178) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * LORIS-MRI tooling configuration (#1170) * add pyproject.toml * keep flake8 (for now) * double type checking * Use Pyright as main type checker * remove comment * test * test * change configuration * add python environment variables * try factorization * test * test * test * test * remove temporary things * test ruff error * ruff output github * further test errors * test pyright output github * test * test * test * change comment * remove test errors * test --------- Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * update docker install (#1171) Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> * add RB override tables for the MRI side (#1173) * Remove php 8.1 and 8.2 from the LORIS tests to keep only test on PHP 8.3 (#1175) * remove php 8.1 and 8.2 testing * update to 8.3 in Dockerfile.test.php8 * ADD minc tools install in gitactions (#1179) * todo install loris mri * test * run pytest in gitaction (#1172) * test * ttt * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * test * Delete test_example.py * rename * done * Delete test/Dockerfile.test.py * test (#1187) * unit tests poc (#1188) * add more pytest global rules (#1194) * Stricter linter configuration (#1193) * stricter ruff config * do not ignore N818 * lint for sorted imports * add python scripts directory (#1196) * Bunch of SQLAlchemy definitions (#1192) * bunch of sqlalchemy definitions * add project orm definition * fix candidate to site query function * optimize docker image (#1195) * fix pytest config (#1197) * fix python scripts (#1199) * remove mysql-connector (#1200) * Improve subject configuration structure (#1150) * refacor subject config * remove suspicious phantom code * change naming * fix rebase lints * cleaner database url * Cecile CandID path join bug fix Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> --------- Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * Integration test to check that the ORM definitions match the LORIS SQL database (#1198) * orm sync integration test * add docker health check * use integration config file * test error * Revert "test error" This reverts commit d1a70e6. * Clean up Docker database scripts (#1202) * clean docker database * remove docker warning * remove unused attribute (#1201) * try fix perl ci * Add file, file_parameter and scanner SQLAlchemy models (#1207) * add tables * rename file model * rename parameter_file * fix lints (#1210) * Unify LORIS-MRI logging (#1191) * new env * change print order in database log * create notification type if needed * fix typing pyright update * Integration test for `run_dicom_archive_loader.py` (#1203) add integration test put keys in github repo debugging add database check add comments and debugging remove s3 keys change converter in test check file tree fix doc fix docs 2 rename variable go back to using s3 keys fix path try fix * rename database sub-namepaces (#1214) * fix warnings (#1215) * Factorize get subject session (#1190) * factorize subject session * re-add site log * Migrate "get all sites from the database" to the SQLAlchemy abstraction (#1216) * Fix new PEP585 Ruff lint (#1218) * Clean-up database scan type (#1139) Complementary PR of LORIS #9304 * Migrate DICOM archive files to new database abstraction (#1217) * wip * use self.dicom_archive.id instead of self.tarchive_id * fix todo * Fix SQLAlchemy file model (#1221) * Improve Python requirements order (#1222) * add python documentation (#1223) * Update SQLAlchemy bindings to match LORIS#9556 (#1229) * trigger_tests * try update * fix old queries for integration tests * Migrate `ImagingUpload` and `MriUploadDB` to the new database abstraction (#1224) * migrate mri upload to sqlalchemy * fix concurrency bug * fix clean up * display mri upload ids use dicom archive id instead of tarchiveid because it is prettier ruff check * migrate scanner to new abstraction (#1232) * Fix install script (#1234) * fix install script and move files related to install into the install directory * fix install script and move files related to install into the install directory * fix install script and move files related to install into the install directory * fix paths in imaging_install_test.sh * fix paths in imaging_install_test.sh * fix SQL error in install script * fix tests * fix tests * fix github action * add cpanfile * copy cpanfile in MRI dockerfile * add new dependency in order to be able to source the Digest::BLAKE2 in the cpanfile * fix Digest::BLAKE2 installation though cpanfile * move config templates to a templates directory and requirements files into a requirements directory * remove copy * fix test * fix test * fix test * fix test * Maxime's feedback * specify python version (#1237) * fix large file hash memory (#1236) * add lib.util module (#1238) * New DICOM study import script (#1117) * rewrite dicom archive * update * remove --year and --target options * get session from config file * remove unused upload argument * fix little oopsie for --session * handle large files hash * fix unhandled modalities * use new lib.util module * Add support to be able to read headers from enhanced DICOMs (#7) * print * print * print * print * add support for enhanced DICOMs * add support for enhanced DICOMs * rework associate files with series * sort dicom series and files before inserting in the database * handle null scanner --------- Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca> * refactoring of CandID on the non-ORM python side * tests * fix insertion in mri_protocol_violation_scans * alphabetical order is hard sometimes * FIx `get_candidate_id` documentation --------- Co-authored-by: Maxime Mulder <maxime-mulder@outlook.com> Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca> Co-authored-by: Shen <kongtiaowangshen@gmail.com>

kongtiaowang changed the title ~~test~~ integration test Jun 20, 2024

kongtiaowang added the Waiting for another PR to be merged label Jun 25, 2024

kongtiaowang removed the Waiting for another PR to be merged label Jul 4, 2024

cmadjar reviewed Aug 1, 2024

View reviewed changes

.github/workflows/flake8_python_linter.yml Outdated Show resolved Hide resolved

laemtl reviewed Aug 6, 2024

View reviewed changes

.github/workflows/loristest.yml Show resolved Hide resolved

cmadjar reviewed Aug 23, 2024

View reviewed changes

test/RB_SQL/RB_Config.sql Show resolved Hide resolved

kongtiaowang added 18 commits August 23, 2024 13:59

test

3741ce3

main Loris here

9df0773

ignore Loris python

2d5bc4b

clone Loris

4789146

rm loris

fb306ce

override test

913c0fb

override test

e3cbf2f

override RB data

79a7766

fix folder

061e605

test php files

75566ab

test

dd935f3

try

0ecc3ac

test

f110c9f

try install loris mri

da8fb0a

os

3218f29

test

be94716

rm modules

259711f

Update flake8_python_linter.yml

c7570a4

cmadjar force-pushed the 2024-06-20-mri-build-integration-test branch from 64b77ef to c7570a4 Compare August 23, 2024 18:04

cmadjar reviewed Aug 23, 2024

View reviewed changes

cmadjar merged commit c669e2b into aces:main Aug 30, 2024
5 checks passed

cmadjar added this to the 27.0 milestone Aug 30, 2024

cmadjar assigned kongtiaowang Aug 30, 2024

cmadjar added this to the 27.0 milestone Apr 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

integration test #1138

integration test #1138

Uh oh!

kongtiaowang commented Jun 20, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmadjar Aug 23, 2024

Uh oh!

cmadjar Aug 23, 2024

Uh oh!

cmadjar Aug 23, 2024

Uh oh!

kongtiaowang Aug 30, 2024

Uh oh!

Uh oh!

Uh oh!

integration test #1138

integration test #1138

Uh oh!

Conversation

kongtiaowang commented Jun 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmadjar Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

cmadjar Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

cmadjar Aug 23, 2024

Choose a reason for hiding this comment

Uh oh!

kongtiaowang Aug 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kongtiaowang commented Jun 20, 2024 •

edited

Loading