Skip to content

integration test #1138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Aug 30, 2024
Merged

integration test #1138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
3741ce3
test
kongtiaowang Jun 20, 2024
9df0773
main Loris here
kongtiaowang Jun 20, 2024
2d5bc4b
ignore Loris python
kongtiaowang Jun 20, 2024
4789146
clone Loris
kongtiaowang Jun 26, 2024
fb306ce
rm loris
kongtiaowang Jun 26, 2024
913c0fb
override test
kongtiaowang Jun 26, 2024
e3cbf2f
override test
kongtiaowang Jun 26, 2024
79a7766
override RB data
kongtiaowang Jul 3, 2024
061e605
fix folder
kongtiaowang Jul 3, 2024
75566ab
test php files
kongtiaowang Jul 3, 2024
dd935f3
test
kongtiaowang Jul 3, 2024
0ecc3ac
try
kongtiaowang Jul 3, 2024
f110c9f
test
kongtiaowang Jul 3, 2024
da8fb0a
try install loris mri
kongtiaowang Jul 3, 2024
3218f29
os
kongtiaowang Jul 3, 2024
be94716
test
kongtiaowang Jul 3, 2024
259711f
rm modules
kongtiaowang Jul 4, 2024
c7570a4
Update flake8_python_linter.yml
kongtiaowang Jul 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions .github/workflows/flake8_python_linter.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,19 @@ jobs:

flake8-lint:
runs-on: ubuntu-latest

strategy:
matrix:
python-version: ["3.11", "3.12"]

steps:
- name: Check out source repository
uses: actions/checkout@v2

- name: Set up Python environment
uses: actions/setup-python@v5
with:
python-version: "3.11"

python-version: ${{ matrix.python-version }}
- name: Flake8 Lint
uses: reviewdog/action-flake8@v3
with:
Expand All @@ -34,5 +37,6 @@ jobs:
# E272: Multiple spaces before keyword
flake8_args: "--ignore=W503,E202,E203,E221,E241,E251,E272,E126,E131,E121,E111,E114,E127
--max-line-length 120
--exclude python/react-series-data-viewer/protocol_buffers/"
--exclude python/react-series-data-viewer/protocol_buffers/
--exclude Loris "
github_token: ${{ secrets.GITHUB_TOKEN }}
92 changes: 92 additions & 0 deletions .github/workflows/loristest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: LORIS Test Suite

on:
- push
- pull_request

jobs:
docker:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
testsuite: ['integration']
php: ['8.1','8.2', '8.3']
ci_node_index: [0]

include:
# add a variable but do not display it in the job's name
- ci_node_total: 1
steps:
- uses: actions/checkout@v3

- name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
extensions: zip, php-ast


- name: Clone Loris repo
run: git clone https://github.com/aces/Loris.git

- name: Override Test Files
run: |
cp test/Dockerfile.test.php8 Loris/
cp test/docker-compose.yml Loris/
cp test/phpunit.xml Loris/test/
cp test/RB_SQL/* Loris/raisinbread/RB_files/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we select here the RB SQL tables we want to source? Technically, we only need the following tables:

  • Config
  • ConfigSettings
  • ImagingFileTypes
  • Project
  • Visit_Windows
  • all the bids-* tables
  • candidate
  • cohort
  • mri_protocol_* tables
  • mri_scan_type
  • parameter_type (however, this could be populated automatically when running the pipeline)
  • permissions* (I would guess this would be needed when using the API)
  • project_cohort_rel
  • psc
  • session
  • user_perm_rel
  • user_project_rel
  • user_psc_rel
  • users
  • visit
  • visit_project_cohort_rel

Purposely skipping the following tables:

  • feedback_mri_* (don't need feedbacks for LORIS-MRI)
  • files (would use a very lightweight version of this table)
  • files_intermediary
  • files_qcstatus (QC so not really used by LORIS-MRI)
  • electrophysiology_uploader (not sure if this is needed)
  • hed_schema (not sure if this has data or has set up information, would need @jeffersoncasimir to decide)
  • hed_schema_nodes (same, @jeffersoncasimir)
  • hrrt_* (empty tables in RB)
  • mri_parameter_form ( I don't think there is any use of that table for LORIS-MRI)
  • mri_processing_protocol (empty table in RB)
  • mri_scanner (has scanner data but the pipeline is supposed to create that if it does not exist so maybe we can ensure the logic of creating the scanner works if we skip this table?)
  • mri_upload* (could have a lightweight version of it or keep it empty)
  • mri_violations_log (could have a lightweight version of it or keep it empty)
  • notification_spool (we don't really need all of that data)
  • parameter_files (could have a lightweight version of it or keep it empty)
  • physiological* tables (could have a lightweight version of it or keep it empty and the script will populate those)
  • server_processes (could probably keep that empty)
  • tarchive* (would have a lightweight version of it with a couple of files)
  • violations_resolved (could have a lightweight version of it or keep it empty)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, you can do it override any files under the RB_SQL folder.


- name: Validate composer.json and composer.lock
run: |
cd Loris
composer validate

- name: Cache Composer packages
id: composer-cache
run: |
cd Loris
echo "::set-output name=dir::$(composer config cache-files-dir)"

- name: Composer cache
uses: actions/cache@v3
with:
path: ${{ steps.composer-cache.outputs.dir }}
key: ${{ runner.os }}-composer-${{ hashFiles('Loris/composer.lock') }}
restore-keys: ${{ runner.os }}-composer-

- name: Change PHP Version in Dockerfile
run: |
cd Loris
sed -i "s/8.0/${{ matrix.php }}/g" Dockerfile.test.php8

- name: Install package dependencies
run: |
sudo apt install -y imagemagick-6-common libmagickcore-6.q16-6 libmagickwand-6.q16-6 \
libprotobuf-dev libprotobuf23 libprotoc23 protobuf-compiler
cd Loris/modules/electrophysiology_browser/jsx/react-series-data-viewer/
protoc protocol-buffers/chunk.proto --js_out=import_style=commonjs,binary:./src/

- name: Install composer dependencies
if: steps.composer-cache.outputs.cache-hit != 'true'
run: |
cd Loris
composer install --prefer-dist --no-progress --no-suggest

- name: Build LORIS
run: |
cd Loris
make dev

- name: Run Test Suite
run: |
# set sandbox to 1 before running the tests
# since some tests only run in sandbox environments
sed -i 's/<sandbox>0<\/sandbox>/<sandbox>1<\/sandbox>/g' Loris/test/config.xml
cd Loris
npm run tests:${{ matrix.testsuite }}
env:
# Specifies how many jobs you would like to run in parallel,
CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
# Use the index from matrix as an environment variable
CI_NODE_INDEX: ${{ matrix.ci_node_index }}
34 changes: 34 additions & 0 deletions test/Dockerfile.test.php8
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
FROM php:8.1

RUN cat /etc/os-release

RUN apt-get update && \
apt-get install -y mariadb-client libzip-dev

RUN docker-php-ext-install pdo_mysql && \
docker-php-ext-install zip

# todo install loris-mri here

# Update the package list and install build-essential, checkinstall, and cmake
RUN apt-get update && \
apt-get install -y build-essential checkinstall cmake

# Install Perl and CPAN
RUN apt-get install -y perl && \
apt-get install -y libterm-readline-gnu-perl && \
apt-get install -y perl-doc && \
apt-get install -y libssl-dev && \
apt-get install -y liblwp-protocol-https-perl && \
cpan CPAN

# Update the package list and install required packages

RUN apt-get update && \
apt-get install -y \
python3 \
python3-dev \
python3-pip \
libmariadb-dev-compat \
libmariadb-dev \
virtualenv
116 changes: 116 additions & 0 deletions test/RB_SQL/RB_Config.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE TABLE `Config`;
LOCK TABLES `Config` WRITE;
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (1,2,'1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (2,3,'LORIS Demonstration Database');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (3,42,'<h3>Example Study Description</h3>\r\n <p>This is a sample description for this study, because it is a new LORIS install that has not yet customized this text.</p>\r\n <p>A LORIS administrator can customize this text in the configuration module, under the configuration option labeled \"Study Description\"</p>\r\n <h3>Useful Links</h3>\r\n <ul>\r\n <li><a href=\"https://github.com/aces/Loris\" >LORIS GitHub Repository</a></li>\r\n <li><a href=\"https://github.com/aces/Loris/wiki/Setup\" >LORIS Setup Guide</a></li>\r\n <li><a href=\"https://www.youtube.com/watch?v=2Syd_BUbl5A\" >A video of a loris on YouTube</a></li>\r\n </ul>');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (4,4,'images/LORIS_logo.png');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (5,5,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (6,6,'5');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (7,7,'99');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (9,9,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (10,10,'2016');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (11,11,'2028');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (12,13,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (13,14,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (15,16,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (16,20,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (17,21,'true');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (18,22,'true');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (19,23,'0');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (20,24,'Customize this text with your citation policy (via Configuration module)');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (22,12,'Ymd');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (23,27,'/data-raisinbread/');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config 27 is there @laemtl

INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (24,28,'/var/www/loris/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (26,30,'/PATH/TO/EXTERNAL/LIBRARY/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (28,32,'/data/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (29,33,'tools/logs/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (30,34,'/data/incoming/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (31,35,'/data/demo/bin/mri/');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and config 35 is here @laemtl

INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (32,36,'/data/incoming/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (33,37,'/PATH/TO/Genomic-Data/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (34,38,'/data/uploads/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (37,44,'localhost');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (39,48,'This database provides an on-line mechanism to store both imaging and behavioural data collected from various locations. Within this framework, there are several tools that will make this process as efficient and simple as possible. For more detailed information regarding any aspect of the database, please click on the Help icon at the top right. Otherwise, feel free to contact us at the DCC. We strive to make data collection almost fun.');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (40,51,'[a-zA-Z]{3}[0-9]{4}_[0-9]{6}_[vV][0-9]+');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (41,52,'(?i).');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (42,53,'(?i)phantom');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (43,54,'(?i)phantom');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (44,55,'true');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (45,56,'t1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (46,56,'t2');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (47,58,'radiology_review');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (48,58,'mri_parameter_form');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (49,60,'no-reply@example.com');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (50,61,'no-reply@example.com');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (51,62,'Produced by LorisDB');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (52,66,'S3cret');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (53,71,'demo');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (54,72,'yourname@example.com');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (55,73,'/data/demo/bin/mri/dicom-archive/get_dicom_info.pl');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (56,74,'1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (57,75,'1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (58,76,'dcm2mnc');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (59,77,'/data-raisinbread/tarchive/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (60,78,'PatientName');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (61,79,'1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (62,80,'1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (64,82,'65');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (65,83,'t1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (66,84,'19');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (67,85,'/opt/niak-0.6.4.1/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (68,86,'INTERLACE_outputDWIFileNameSuffix');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (69,18,'aosi');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (70,18,'bmi');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (72,18,'medical_history');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (73,18,'mri_parameter_form');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (74,18,'radiology_review');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (75,87,'/issue_tracker');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (76,88,'localizer');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (77,88,'scout');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (78,89,'true');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (79,90,'0');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (80,91,'mri_parameter_form');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (82,92,'prod');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (83,94,'t1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (84,95,'flair');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (85,95,'t1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (86,95,'t2');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (87,95,'pd');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (88,96,'flair');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (89,96,'t1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (90,96,'t2');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (91,96,'pd');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (92,97,'/data/publication_uploads/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (93,98,'/data/publication_uploads/to_be_deleted/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (94,99,'true');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (95,49,'1000');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (96,100,'environment');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (97,70,'/data-raisinbread/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (98,93,'V1');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (99,101,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (102,19,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (103,102,'/data/document_repository_uploads/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (104,103,'/data/data_release_uploads/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (105,104,'Ymd');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (106,105,'Y-m-d H:i:s');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (107,106,'/data/issue_tracker/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (108,107,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (113,112,'warning');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (114,113,'warning');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (115,114,'none');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (116,116,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (117,117,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (118,118,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (119,119,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (120,120,'database_config.py');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (121,125,'false');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (122,126,'0');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (123,127,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (124,128,'');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (126,129,'365');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (127,130,'/var/www/loris/');
INSERT INTO `Config` (`ID`, `ConfigID`, `Value`) VALUES (128,131,'/data/EEGUploadIncomingPath/');

UNLOCK TABLES;
SET FOREIGN_KEY_CHECKS=1;
Loading
Loading