Skip to content

Commit 5af5068

Browse files
authored
Merge pull request #176 from mcode/develop
v2.0.0
2 parents 58af417 + bb4d1fb commit 5af5068

File tree

114 files changed

+112787
-2839
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

114 files changed

+112787
-2839
lines changed

.github/workflows/ci-workflow.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ on: [push, pull_request]
44

55
jobs:
66
lint:
7-
name: Check lint
7+
name: Lint
88
runs-on: ubuntu-latest
99
steps:
1010
- uses: actions/checkout@v1
@@ -16,7 +16,7 @@ jobs:
1616
env:
1717
CI: true
1818
test:
19-
name: Run tests on ${{ matrix.os }}
19+
name: Tests on ${{ matrix.os }}
2020
runs-on: ${{ matrix.os }}
2121
strategy:
2222
matrix:

README.md

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# mCODE Extraction Framework
22

3-
A Node.js framework for extracting mCODE FHIR resources. All resources are profiled per the [mCODE v1.0.0 R4 FHIR Implementation Guide](https://mcodeinitiative.github.io/index.html)
3+
A Node.js framework for extracting mCODE FHIR resources. All resources are profiled per the [mCODE v1.16.0 R4 FHIR Implementation Guide](http://hl7.org/fhir/us/mcode/2021May/)
44

55
## Table of Contents
66

@@ -29,7 +29,8 @@ A Node.js framework for extracting mCODE FHIR resources. All resources are profi
2929

3030
## Prerequisites
3131

32-
- [Node.js >= 12](https://nodejs.org/en/)
32+
- [Node.js v14.18.1 (LTS as of Oct 2021)](https://nodejs.org/en/)
33+
- npm v6.14.15 (included in LTS of Oct 2021)
3334

3435
### Latest
3536

@@ -43,7 +44,7 @@ There are various extractors, modules, and helper functions exposed by the frame
4344

4445
Each extractor exposes an asynchronous `get` function that will return a bundle of extracted mCODE data.
4546

46-
``` JavaScript
47+
```JavaScript
4748
// Example
4849
const { CSVCancerDiseaseStatusExtractor, logger} = require('mcode-extraction-framework');
4950

@@ -91,8 +92,11 @@ Examples files for these extractor can be found in the [`test/sample-client-data
9192

9293
After exporting your CSV files to the `data` directory, kickstart the creation of a configuration file by renaming the provided `csv.config.example.json` to `csv.config.json`. Then, ensure the following configuration parameters are properly set:
9394

94-
1. `patientIdCsvPath` should provide a file path to a CSV file containing MRN's for relevant patients;
95-
2. For each extractor, `filePath:` should provide a file path to a CSV file containing that corresponding extractor's data;
95+
1. `patientIdCsvPath` should correspond to an absolute file path to a CSV file containing MRN's for relevant patients;
96+
2. `commonExtractorArgs.dataDirectory` should correspond to an absolute path to the dataDirectory containing all your exported CSV files;
97+
3. For each extractor, `fileName` should correspond to the file name this extractor should be reading from. Note: combining the `dataDirectory` above and `fileName` should resolve to a file on disk containing this corresponding extractor's data;
98+
99+
**Note**: Previous versions of the MEF suggested using a `filePath` property for each extractor; while this property should still work without issue, the recommended approach is to use a common dataDirectory for all CSV files and to have each Extractor call out the name of the CSV file they need.
96100

97101
For instructions on setting up an email notification trigger whenever an error is encountered in extraction, see the [Email Notification](#Email-Notification) section below.
98102

@@ -115,7 +119,7 @@ In order to send an email, users must specify the hostname or IP address of an S
115119
- `port`: `<number>` (Optional) The port to connect to (defaults to 587)
116120
- `to`: `<string[]>` Comma separated list or an array of recipients email addresses that will appear on the _To:_ field
117121
- `from`: `<string>` (Optional) The email address of the sender. All email addresses can be plain `'sender@server.com'` or formatted `'"Sender Name" sender@server.com'` (defaults to mcode-extraction-errors@mitre.org, which cannot receive reply emails)
118-
- `tlsRejectUnauthorized`: `<boolean>` (Optional) A boolean value to set the [node.js TLSSocket option](https://nodejs.org/api/tls.html#tls_class_tls_tlssocket) for rejecting any unauthorized connections, `tls.rejectUnauthorized`. (defaults to `true`)
122+
- `tlsRejectUnauthorized`: `<boolean>` (Optional) A boolean value to set the [node.js TLSSocket option](https://nodejs.org/api/tls.html#tls_class_tls_tlssocket) for rejecting any unauthorized connections, `tls.rejectUnauthorized`. (defaults to `true`)
119123

120124
An example of this object can be found in [`config/csv.config.example.json`](config/csv.config.example.json).
121125

@@ -134,20 +138,33 @@ npm start -- --entries-filter --from-date YYYY-MM-DD --to-date YYY-MM-DD --run-l
134138
### Masking Patient Data
135139

136140
Patient data can be masked within the extracted `Patient` resource. When masked, the value of the field will be replaced with a [Data Absent Reason extension](https://www.hl7.org/fhir/extension-data-absent-reason.html) with the code `masked`.
137-
Patient properties that can be masked are: `gender`, `mrn`, `name`, `address`, `birthDate`, `language`, `ethnicity`, `birthsex`, and `race`.
141+
Patient properties that can be masked are: `genderAndSex`, `mrn`, `name`, `address`, `birthDate`, `language`, `ethnicity`, `race`, `telecom`, `multipleBirth`, `photo`, `contact`, `generalPractitioner`, `managingOrganization`, and `link`.
138142
To mask a property, provide an array of the properties to mask in the `constructorArgs` of the Patient extractor. For example, the following configuration can be used to mask `address` and `birthDate`:
139143

140144
```bash
141145
{
142146
"label": "patient",
143147
"type": "CSVPatientExtractor",
144148
"constructorArgs": {
145-
"filePath": "./data/patient-information.csv"
149+
"fileName": "patient-information.csv"
146150
"mask": ["address", "birthDate"]
147151
}
148152
}
149153
```
150154

155+
Alternatively, providing a string with a value of `all` in the `constructorArgs` of the Patient extractor will mask all of the supported properties listed above. The following configuration can be used to mask all properties of the `Patient` resource, rather than listing each individual property:
156+
157+
```bash
158+
{
159+
"label": "patient",
160+
"type": "CSVPatientExtractor",
161+
"constructorArgs": {
162+
"fileName": "patient-information.csv"
163+
"mask": "all"
164+
}
165+
}
166+
```
167+
151168
### Extraction Date Range
152169

153170
The mCODE Extraction Client will extract all data that is provided in the CSV files by default, regardless of any dates associated with each row of data. It is recommended that any required date filtering is performed outside of the scope of this client.
@@ -182,6 +199,22 @@ cat -v <file.csv>
182199

183200
If there is an unexpected symbol at the beginning of the file, then there may be a byte order marker that needs to be removed.
184201

202+
#### Troubleshooting Additional Errors
203+
The mCODE Extraction Framework uses the node `csv-parse` library to parse specified CSV files. [Parsing options for the `csv-parse` library](https://csv.js.org/parse/options/) can be included in the configuration file within the `commonExtractorArgs.csvParse.options` section. For example, the following configuration will pass the `to` option to the `csv-parse` module, causing the mCODE Extraction Framework to only read CSV files up to the specified line number:
204+
205+
```
206+
"commonExtractorArgs": {
207+
"dataDirectory": "/Users/*****/Documents/dataDirectory",
208+
"csvParse": {
209+
"options": {
210+
"to": 3
211+
}
212+
}
213+
},
214+
```
215+
216+
**Note:** The mCODE Extraction Framework enables the `bom`, `skip_empty_lines`, and `skip_lines_with_empty_values` options by default, including these options in the configuration file will cause these default options to be overwritten.
217+
185218
## Terminology and Architecture
186219

187220
This framework consists of three key components: Extractors, Modules and Templates. Below is, in order:

config/csv.config.example.json

Lines changed: 31 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
{
2-
"patientIdCsvPath": "./data/patient-mrns.csv",
3-
"commonExtractorArgs": {},
2+
"patientIdCsvPath": "Users/YourAccount/absolute/path/to/patient-mrns.csv",
3+
"commonExtractorArgs": {
4+
"dataDirectory": "Users/YourAccount/absolute/path/to/data/directory"
5+
},
46
"notificationInfo": {
57
"host": "smtp.example.com",
68
"port": 587,
@@ -16,72 +18,86 @@
1618
"label": "patient",
1719
"type": "CSVPatientExtractor",
1820
"constructorArgs": {
19-
"filePath": "./data/patient-information.csv"
21+
"fileName": "patient-information.csv"
2022
}
2123
},
2224
{
2325
"label": "condition",
2426
"type": "CSVConditionExtractor",
2527
"constructorArgs": {
26-
"filePath": "./data/condition-information.csv"
28+
"fileName": "condition-information.csv"
2729
}
2830
},
2931
{
3032
"label": "cancerDiseaseStatus",
3133
"type": "CSVCancerDiseaseStatusExtractor",
3234
"constructorArgs": {
33-
"filePath": "./data/cancer-disease-status-information.csv"
35+
"fileName": "cancer-disease-status-information.csv"
3436
}
3537
},
3638
{
3739
"label": "clinicalTrialInformation",
3840
"type": "CSVClinicalTrialInformationExtractor",
3941
"constructorArgs": {
40-
"filePath": "./data/clinical-trial-information.csv",
42+
"fileName": "clinical-trial-information.csv",
4143
"clinicalSiteID": "example-site-id"
4244
}
4345
},
4446
{
4547
"label": "treatmentPlanChange",
4648
"type": "CSVTreatmentPlanChangeExtractor",
4749
"constructorArgs": {
48-
"filePath": "./data/treatment-plan-change-information.csv"
50+
"fileName": "treatment-plan-change-information.csv"
4951
}
5052
},
5153
{
5254
"label": "staging",
5355
"type": "CSVStagingExtractor",
5456
"constructorArgs": {
55-
"filePath": "./data/staging-information.csv"
57+
"fileName": "staging-information.csv"
58+
}
59+
},
60+
{
61+
"label": "cancerRelatedMedicationAdministration",
62+
"type": "CSVCancerRelatedMedicationAdministrationExtractor",
63+
"constructorArgs": {
64+
"fileName": "cancer-related-medication-administration-information.csv"
5665
}
5766
},
5867
{
59-
"label": "cancerRelatedMedication",
60-
"type": "CSVCancerRelatedMedicationExtractor",
68+
"label": "cancerRelatedMedicationRequest",
69+
"type": "CSVCancerRelatedMedicationRequestExtractor",
6170
"constructorArgs": {
62-
"filePath": "./data/cancer-related-medication-information.csv"
71+
"fileName": "cancer-related-medication-request-information.csv"
6372
}
6473
},
6574
{
6675
"label": "genericObservations",
6776
"type": "CSVObservationExtractor",
6877
"constructorArgs": {
69-
"filePath": "./data/observation-information.csv"
78+
"fileName": "observation-information.csv"
7079
}
7180
},
7281
{
7382
"label": "genericProcedures",
7483
"type": "CSVProcedureExtractor",
7584
"constructorArgs": {
76-
"filePath": "./data/procedure-information.csv"
85+
"fileName": "procedure-information.csv"
7786
}
7887
},
7988
{
8089
"label": "adverseEvent",
8190
"type": "CSVAdverseEventExtractor",
8291
"constructorArgs": {
83-
"filePath": "./test/sample-client-data/adverse-event-information.csv"
92+
"fileName": "adverse-event-information.csv"
93+
}
94+
},
95+
{
96+
"label": "ctcAdverseEvent",
97+
"type": "CSVCTCAdverseEventExtractor",
98+
"constructorArgs": {
99+
"fileName": "ctc-adverse-event-information.csv"
84100
}
85101
}
86102
]
87-
}
103+
}

docs/CSV_Templates.xlsx

100644100755
629 KB
Binary file not shown.
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
mrn,requestId,code,codeSystem,displayText,treatmentReasonCode,treatmentReasonCodeSystem,treatmentReasonDisplayText,procedureIntent,status,intent,authoredOn,requesterId
2+
mrn-1,requestId-1,code,code-system,code-text,code,code-system,display-text,procedure-intent-code,status-code,intent-code,YYY-MM-DD,requesterId-1
3+
mrn-2,requestId-2,code,code-system,code-text,code,code-system,display-text,procedure-intent-code,status-code,intent-code,YYY-MM-DD,requesterId-2

docs/ctc-adverse-event.csv

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
mrn,adverseEventId,adverseEventCode,adverseEventCodeSystem,adverseEventCodeVersion,adverseEventDisplayText,adverseEventText,suspectedCauseId,suspectedCauseType,seriousness,seriousnessCodeSystem,seriousnessDisplayText,category,categoryCodeSystem,categoryDisplayText,studyId,effectiveDate,recordedDate,grade,expectation,resolvedDate,seriousnessOutcome,actor,functionCode
2+
mrn-full-example,example-id-1,event-code,code-system,code-version,code-display,event-text,cause-id,resourceType,seriousness-code,code-system,seriousness-display,category-code,code-system,category-dislpay,id,1994-12-09,1994-12-09,1,expectation-code,YYYY-MM-DD,seriousness-outcome-code,practitioner-id,function-code
3+
mrn-two-category-example,example-id-2,event-code,code-system,code-version,code-display,event-text,cause-id,resourceType,seriousness-code,code-system,seriousness-display,category-code,code-system,category-dislpay,id,1994-12-09,1994-12-09,1,expectation-code,YYYY-MM-DD,seriousness-outcome-code,practitioner-id,
4+
mrn-minimal-example,,code-from-default-system,,,,,,,,,,,,,1994-12-09,,1,,,,,,

0 commit comments

Comments
 (0)