FHIR References are altered/invalid

[jawalonoski]

Reported by HSPC

1. https://syntheticmass.mitre.org/fhir/Patient - to get 100 patients
2. https://syntheticmass.mitre.org/fhir/Patient//$everything - on every patient to get each patient's set of resources.
3. For those results, I build a FHIR transaction bundle to import the set of patient & patient resources as a transaction

Out of the 100 patients, only 9 import successfully in to my FHIR v1.8 server. However, if I pull patients from a server of mine where I previously pushed 2000+ patients generated with Synthea, I have full success. (When I previously imported the 2000+ patients, I saw no errors.)

I'm getting the following types of errors:

For Patient/58b3663f3425def0f0f6af86

Failure during REST processing: ca.uhn.fhir.rest.server.exceptions.UnprocessableEntityException: Invalid reference found at path 'DiagnosticReport.performer'. Resource type 'MedicationRequest' is not valid for this path
Response Status : HTTP/1.1 422 Unprocessable Entity .
Response Detail :{
  "resourceType": "OperationOutcome",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Invalid reference found at path 'DiagnosticReport.performer'. Resource type 'MedicationRequest' is not valid for this path</pre></td>\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
  },
  "issue": [
    {
      "severity": "error",
      "code": "processing",
      "diagnostics": "Invalid reference found at path 'DiagnosticReport.performer'. Resource type 'MedicationRequest' is not valid for this path"
    }
  ]
}

AND

For Patient/58b3663f3425def0f0f6ae67

Failure during REST processing: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: Resource Encounter/58b3663f3425def0f0f6aeb0 not found, specified in path: Condition.context
Response Status : HTTP/1.1 400 Bad Request .
Response Detail :{
  "resourceType": "OperationOutcome",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Resource Encounter/58b3663f3425def0f0f6aeb0 not found, specified in path: Condition.context</pre></td>\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
  },
  "issue": [
    {
      "severity": "error",
      "code": "processing",
      "diagnostics": "Resource Encounter/58b3663f3425def0f0f6aeb0 not found, specified in path: Condition.context"
    }
  ]
}

[jawalonoski]

Somehow the FHIR server is rewriting our references.

For Patient/58b3663f3425def0f0f6af86

Here is the offending DiagnosticReport

    {
      "resource": {
        "resourceType": "DiagnosticReport",
        "id": "58b3663f3425def0f0f6af8b",
        "status": "final",
        "code": {
          "coding": [
            {
              "system": "http://loinc.org",
              "code": "69409-1",
              "display": "U.S. standard certificate of death - 2003 revision"
            }
          ]
        },
        "subject": {
          "reference": "Patient/58b3663f3425def0f0f6af86"
        },
        "encounter": {
          "reference": "Encounter/58b3663f3425def0f0f6af87"
        },
        "effectiveDateTime": "1999-09-18T06:51:52-04:00",
        "issued": "1999-09-18T06:51:52-04:00",
        "performer": [
          {
            "reference": "MedicationRequest/58b3663f3425def0f0f6af92"
          }
        ],
        "result": [
          {
            "reference": "Observation/58b3663f3425def0f0f6af8a"
          }
        ]
      },
      "search": {
        "mode": "include"
      }
    }

According the Synthea source code: the DiagnosticReport.performer field should look like this:

'performer' => [{ 'display' => 'Hospital Lab' }])

See https://github.com/synthetichealth/synthea/blob/master/lib/records/fhir.rb#L305

[cmoesel]

Interesting. It would be odd for the FHIR server to be doing that -- as the serialization parameters for DiagnosticReport are generated from the spec, and are pretty clearly pulling from the performer field (see: https://github.com/synthetichealth/gofhir/blob/master/vendor/github.com/intervention-engine/fhir/models/diagnosticreport.go#L46).

Does Synthea maintain a local cache of the FHIR records it posts to the server? Perhaps it is the FHIR server, but it would be good to confirm the actual data that is being posted first (if that's easy).

[jawalonoski]

Maybe the bulkloader then?

I didn't check the raw file for that Patient (yet) but running Synthea doesn't show any results like that. In fact, there is no code within Synthea that sets any field named performer except the line of code that I linked to.

[dehall]

Re: the DiagnosticReport - Per the FHIR spec, DiagnosticReport.performer is a Reference to a Practitioner|Organization, but in Synthea it's being exported directly within the DiagnosticReport (not sure of the proper FHIR terminology). Maybe the server, expecting a Reference, winds up picking a reference to something else somehow?

I can update Synthea so that it correctly exports a Reference (or nothing, since the performer field is optional). If it's possible to modify the live data to clear out the existing performer field on all DiagnosticReports that might be ideal so that the entire data set doesn't have to be regenerated.

Re: the "Resource Encounter not found, specified in path: Condition.context" issue, Synthea is correctly using a reference there, so that could still be an issue with the FHIR server.

[jawalonoski]

@dehall Let's not modify Synthea yet.

First, let's check the raw input file for Patient/58b3663f3425def0f0f6af86 which has Synthea Identifier of 003a51c1-37cb-407c-9396-b638a18dfc36

[cmoesel]

@dehall I think that what Synthea is doing is valid (although probably not ideal). { 'display' => 'Hospital Lab' } does follow the form of a reference (see: http://hl7.org/fhir/2017Jan/references.html). That said, a reference with just a display is not very helpful, and doesn't even provide enough context to programmatically determine if it's meant to be Practitioner reference or an Organization reference.

If it turns out the data Synthea creates is fine (and it's starting to look that way), then the bulk loader might be a good place to start if that's how it's getting into the DB. I forgot you weren't just posting them to the FHIR API.

[dehall]

You're right, Synthea is creating a Reference with no 'reference' field; Practitioner and Observation don't have the 'display' field.

[cjduffett]

The bulk loader does modify references here:
https://github.com/synthetichealth/bulkfhirloader/blob/master/bulkloader/bundle.go#L127-L147

The Performer attribute of DiagnosticReport is of type []Reference, but the bulk loader seems to only modify references starting with cid:, which would prevent it from modifying Performer.

[cjduffett]

Regarding the DiagnosticReport bug:

As a temporary fix, I'm going to remove the performer field for all DiagnosticReports in the data set. This bug is most likely due to our bulkloader rewriting references on load, but I've yet to find the actual culprit.

Regarding the Encounter bug:

The encounter resource being referenced (58b3663f3425def0f0f6aeb0) exists but contains only the id and resourceType fields:

{
  "resourceType": "Encounter",
  "id": "58b3663f3425def0f0f6aeb0"
}

This encounter is critically missing it's patient reference, which will cause issues when building the $everything query. While I'm not certain why this resource is missing the remaining fields, this typically occurs when the server fails to parse malformed JSON in a request. The JSON for this encounter resource likely had a parse error after the id and resourceType fields were processed, so the server stopped there.