Skip to content

Conversation

@mikix
Copy link
Contributor

@mikix mikix commented Sep 16, 2025

Checklist

  • Consider if documentation (like in docs/) needs to be updated
  • Consider if tests should be added

@mikix mikix force-pushed the mikix/aux-resources branch from e7c3ac5 to 6b47cef Compare September 16, 2025 15:19
@mikix mikix changed the title Add more resources Support Location, Organization, Practitioner, and PractitionerRole Sep 16, 2025
@mikix mikix force-pushed the mikix/aux-resources branch from 6b47cef to 8a110d7 Compare September 16, 2025 15:20
class BaseCovidCtakesTask(tasks.BaseNlpTask):
"""Covid Symptom study task, to generate symptom lists from ED notes using cTAKES + a polarity check"""

tags: ClassVar = {"covid_symptom", "gpu"}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tags cleanup is unrelated - I just noticed it while adding new tasks.

I had made these tags obsolete when I added the nlp subtask and took out the no-longer-needed --task-filter support. So this is just some cleanup around that.


ARG ETL_VERSION
RUN [ -z "$ETL_VERSION" ] || sed -i "s/0\.0\.0/$ETL_VERSION/" /app/cumulus_etl/__init__.py
RUN [ -z "$ETL_VERSION" ] || sed -i "s/1\!0\.0\.0/$ETL_VERSION/" /app/cumulus_etl/__init__.py
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 1! prefix was slipping into the Docker releases, so this just updates the regex to chop that off.


// ** Location: https://www.hl7.org/fhir/R4/location.html **
// This is not a patient-linked resource, so we aren't as worried about PHI here.
{"path": "Location.identifier.where(system='http://hl7.org/fhir/sid/us-npi')", "method": "keep"},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One interesting thing to call out here - normally, we don't include identifier at all in resources, since it can have business-identifiers that could be PHI. Famously MRNs would qualify, but we do it for all patient-linked resources out of an abundance of caution.

But for these new resources, there's less risk of that. But still, in my sample Epic export, there are tax IDs in there and other Epic-internal IDs. While I guess that's not exactly secret intel either, I figured the standard NPI identifiers would be (should be?) good enough. And even that is a big deal, privacy wise - because for Practitioners below, you can get name etc from that. But practitioners have a different level of privacy expectation here. And matching by NPI could be useful for a study / internal QA / is sometimes the only useful identifier left after we strip out name, telephone, and address.

Open to tweak this, but that's my initial pitch 🤷

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we should kick this conversation up the chain - i buy it, but who knows about edge cases

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For posterity, our team talked about this internally - we figured this approach is fine. 👍

// keep just the year for privacy (note: 90+ HIPAA grouping is done downstream in SQL
"cases": {"true": "$this.toString().replaceMatches('^(?<year>\\\\d+).*', '${year}')"}},
// Skip Practitioner.photo
{"path": "Practitioner.qualification.identifier", "method": "keep"},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another identifier callout - this is identifying the qualification, not a human. So I figured, just allow all identifiers, no filtering here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thought - i buy it, let's check.

@mikix mikix force-pushed the mikix/aux-resources branch from 8a110d7 to 6a9e242 Compare September 16, 2025 15:59
@mikix mikix marked this pull request as ready for review September 16, 2025 15:59
@github-actions
Copy link

github-actions bot commented Sep 16, 2025

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
4368 4326 99% 98% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
cumulus_etl/deid/scrubber.py 100% 🟢
cumulus_etl/etl/studies/covid_symptom/covid_tasks.py 100% 🟢
cumulus_etl/etl/studies/irae/irae_tasks.py 100% 🟢
cumulus_etl/etl/tasks/base.py 100% 🟢
cumulus_etl/etl/tasks/basic_tasks.py 100% 🟢
cumulus_etl/etl/tasks/nlp_task.py 100% 🟢
cumulus_etl/etl/tasks/task_factory.py 100% 🟢
TOTAL 100% 🟢

updated for commit: c9185ef by action🐍


// ** Location: https://www.hl7.org/fhir/R4/location.html **
// This is not a patient-linked resource, so we aren't as worried about PHI here.
{"path": "Location.identifier.where(system='http://hl7.org/fhir/sid/us-npi')", "method": "keep"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we should kick this conversation up the chain - i buy it, but who knows about edge cases

// keep just the year for privacy (note: 90+ HIPAA grouping is done downstream in SQL
"cases": {"true": "$this.toString().replaceMatches('^(?<year>\\\\d+).*', '${year}')"}},
// Skip Practitioner.photo
{"path": "Practitioner.qualification.identifier", "method": "keep"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thought - i buy it, let's check.

@mikix mikix force-pushed the mikix/aux-resources branch from fa685ed to c9185ef Compare September 18, 2025 20:06
@mikix mikix merged commit 7e487d2 into main Sep 18, 2025
3 checks passed
@mikix mikix deleted the mikix/aux-resources branch September 18, 2025 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants