Merge pull request #113 from mcode/develop

Develop

Author: Dtphelan1

README.md

@@ -19,6 +19,8 @@ A Node.js framework for extracting mCODE FHIR resources. All resources are profi
     - [Masking Patient Data](#masking-patient-data)
     - [Extraction Date Range](#extraction-date-range)
       - [CLI From-Date and To-Date (NOT recommended use)](#cli-from-date-and-to-date-not-recommended-use)
+    - [Troubleshooting](#troubleshooting)
+      - [Byte Order Markers in CSV Files](#byte-order-markers-in-csv-files)
   - [Terminology and Architecture](#terminology-and-architecture)
     - [Glossary](#glossary)
     - [High Level Diagram](#high-level-diagram)
@@ -58,13 +60,13 @@ const cancerDiseaseStatusExtractor = new CSVCancerDiseaseStatusExtractor('path-t
 The framework also contains the [MCODEClient](src/client/MCODEClient.js) which has registered all of the extractors in this repo. Once you have exported CSV data and updated your configuration file, use the mCODE Extraction client by running the following:
 
 ```bash
-node src/cli/cli.js [options]
+npm start -- [options]
 ```
 
 To see all the options that can be used with the mCODE client, run the following:
 
 ```bash
-node src/cli/cli.js --help
+npm start -- --help
 ```
 
 ### First Time User Guide
@@ -125,7 +127,7 @@ Whenever the mCODE Extraction Client successfully runs with the `--entries-filte
 Users can specify a different location for the file by using the `--run-log-filepath <path>` CLI option. Users will need to create this file before running the mCODE Extraction Client with `--entries-filter` and a date range. Initially, this file's contents should be an empty array, `[]`. For example:
 
 ```bash
-node src/cli/cli.js --entries-filter --from-date YYYY-MM-DD --to-date YYY-MM-DD --run-log-filepath path/to/file.json
+npm start -- --entries-filter --from-date YYYY-MM-DD --to-date YYY-MM-DD --run-log-filepath path/to/file.json
 ```
 
 ### Masking Patient Data
@@ -158,9 +160,21 @@ If any filtering on data elements in CSV files is required, the `entries-filter`
 If a `from-date` is provided as an option when running the mCODE Extraction Client, it will be used to filter out any data elements that are recorded before that date based on the `dateRecorded` column in the CSV files. If a `to-date` is provided as an option, it will be used to filter out any data elements that are recorded after that date based on the `dateRecorded` column in the CSV files. If no `to-date` is provided, the default is today. If no `from-date` is provided, the mCODE Extraction Client will look to a run log file (details [above](#Logging-Successful-Extractions)) to find the most recent run and use the `to-date` of that run as the `from-date` for the current run, allowing users to only run the extraction on data elements that were not included in previous runs. If there are no previous run times logged, a `from-date` needs to be provided when running the extraction when the `entries-filter` option is provided. If the `entries-filter` option is not provided, any `from-date` and `to-date` options will be ignored, none of the data elements will be filtered by date, and a successful run will not be logged since there is no specified date range. An example running the client with the `from-date` and `to-date` is as follows:
 
 ```bash
-node src/cli/cli.js --entries-filter --from-date <YYYY-MM-DD> --to-date <YYYY-MM-DD> --config-filepath <path>
+npm start -- --entries-filter --from-date <YYYY-MM-DD> --to-date <YYYY-MM-DD> --config-filepath <path>
 ```
 
+### Troubleshooting
+
+#### Byte Order Markers in CSV Files
+
+The extraction client has built-in handling of byte order markers for CSV files in UTF-8 and UTF-16LE encodings. When using CSV files in other encodings, if you experience unexpected errors be sure to check for a byte order marker at the beginning of the file. One way to check is to run the following command from the command line:
+
+```bash
+cat -v <file.csv>
+```
+
+If there is an unexpected symbol at the beginning of the file, then there may be a byte order marker that needs to be removed.
+
 ## Terminology and Architecture
 
 This framework consists of three key components: Extractors, Modules and Templates. Below is, in order:

config/csv.config.example.json

@@ -13,17 +13,17 @@
   },
   "extractors": [
     {
-      "label": "condition",
-      "type": "CSVConditionExtractor",
+      "label": "patient",
+      "type": "CSVPatientExtractor",
       "constructorArgs": {
-        "filePath": "./data/condition-information.csv"
+        "filePath": "./data/patient-information.csv"
       }
     },
     {
-      "label": "patient",
-      "type": "CSVPatientExtractor",
+      "label": "condition",
+      "type": "CSVConditionExtractor",
       "constructorArgs": {
-        "filePath": "./data/patient-information.csv"
+        "filePath": "./data/condition-information.csv"
       }
     },
     {

docs/CSV_Templates.xlsx

@@ -1,4 +1,4 @@
 mrn,adverseEventId,adverseEventCode,adverseEventCodeSystem,adverseEventDisplayText,suspectedCauseId,suspectedCauseType,seriousness,seriousnessCodeSystem,seriousnessDisplayText,category,categoryCodeSystem,categoryDisplayText,severity,actuality,studyId,effectiveDate,recordedDate
 mrn-full-example,example-id-1,event-code,code-system,code-display,cause-id,resourceType,seriousness-code,code-system,seriousness-display,category-code,code-system,category-dislpay,mild,actual,id,1994-12-09,1994-12-09
-mrn-two-category-example,example-id-1,event-code,code-system,code-display,cause-id,resourceType,seriousness-code,code-system,seriousness-display,category-code|category-code,code-system|code-system,category-display|category-display,mild,actual,id,1994-12-09,1994-12-09
+mrn-two-category-example,example-id-2,event-code,code-system,code-display,cause-id,resourceType,seriousness-code,code-system,seriousness-display,category-code|category-code,code-system|code-system,category-display|category-display,mild,actual,id,1994-12-09,1994-12-09
 mrn-minimal-example,,code-from-default-system,,,,,,,,,,,,,,1994-12-09,

docs/adverse-event.csv

@@ -1,3 +1,3 @@
 mrn,observationId,status,code,codeSystem,displayName,value,valueCodeSystem,effectiveDate,bodySite,laterality
-mrn-1,example-id,example-status,example-code,example-system,example-name,example-value,example-system,YYYY-MM-DD,example-site,example-laterality
-mrn-2,example-id,example-status,example-code,example-system,example-name,example-value,example-system,YYYY-MM-DD,example-site,example-laterality
+mrn-1,example-id-1,example-status,example-code,example-system,example-name,example-value,example-system,YYYY-MM-DD,example-site,example-laterality
+mrn-2,example-id-2,example-status,example-code,example-system,example-name,example-value,example-system,YYYY-MM-DD,example-site,example-laterality

docs/observation.csv

@@ -1,3 +1,3 @@
 mrn,procedureId,conditionId,status,code,codeSystem,displayName,reasonCode,reasonCodeSystem,reasonDisplayName,effectiveDate,bodySite,laterality,treatmentIntent
-mrn-1,example-id,example-condition-id,example-status,example-code,example-system,example-name,example-reason-code,example-system,example-reason-display,YYYY-MM-DD,example-site,example-laterality,example-treatment-intent
-mrn-2,example-id,example-condition-id,example-status,example-code,example-system,example-name,example-reason-code,example-system,example-reason-display,YYYY-MM-DD,example-site,example-laterality,example-treatment-intent
+mrn-1,example-id-1,example-condition-id-1,example-status,example-code,example-system,example-name,example-reason-code,example-system,example-reason-display,YYYY-MM-DD,example-site,example-laterality,example-treatment-intent
+mrn-2,example-id-2,example-condition-id-2,example-status,example-code,example-system,example-name,example-reason-code,example-system,example-reason-display,YYYY-MM-DD,example-site,example-laterality,example-treatment-intent

docs/procedure.csv

@@ -1,3 +1,3 @@
-mrn,reasonCode,changed,dateOfCarePlan,dateRecorded
-mrn-1,281647001,true,2020-04-15,2020-05-01
-mrn-2,281647001,true,2020-03-30,2020-05-01
+mrn,reasonCode,reasonDisplayText,changed,dateOfCarePlan,dateRecorded
+mrn-1,281647001,Adverse reaction (disorder),true,2020-04-15,2020-05-01
+mrn-2,,,false,2020-03-30,2020-05-01