Feature/Adding-HTS-data-to-data-exchange-schema

[nobert-mumo]

Jira ticket : https://thepalladiumgroup.atlassian.net/browse/KHP3-8648

Summary by Sourcery

Add third test kit support to HTS pipelines and introduce a new data exchange table for HTS test linelists

New Features:

• Add end-to-end support for third HTS test kit details (name, lot number, expiry date, result) in ETL scripts and dimensions
• Create new DATA_EXCHANGE.dbo.linelist_hts_tests table to combine HTS client tests and test kit information for reporting

Enhancements:

• Include KEPH_Level in the facility dimension load process
• Expose EncounterId in the FactHTSTestKits load script to link source encounters

[sourcery-ai]

Reviewer's Guide

This PR extends the HTS data pipeline by adding support for a third test kit (including its name, lot number, expiry, and result) across ODS intermediate tables, dimension loads, NDWH fact tables, and the data-exchange reporting schema, with appropriate joins into DimTestKitName and DimDate and a new linelist_hts_tests table.

Entity Relationship diagram for updated FactHTSTestKits and DimTestKitName tables

erDiagram
    NDWH_Fact_FactHTSTestKits {
        INT FactKey PK
        INT EncounterId
        INT PatientKey FK
        INT FacilityKey FK
        INT PartnerKey FK
        INT AgencyKey FK
        INT TestKitName1Key FK
        VARCHAR TestKitLotNumber1
        INT TestKitExpiry1DateKey FK
        VARCHAR TestResult1
        INT TestKitName2Key FK
        VARCHAR TestKitLotNumber2
        INT TestKitExpiry2DateKey FK
        VARCHAR TestResult2
        INT TestKitName3Key FK
        VARCHAR TestKitLotNumber3
        INT TestKitExpiry3DateKey FK
        VARCHAR TestResult3
        DATE LoadDate
    }
    NDWH_Dim_DimTestKitName {
        INT TestKitNameKey PK
        VARCHAR TestKitName
    }
    NDWH_Dim_DimDate {
        INT DateKey PK
        DATE Date
    }
    NDWH_Fact_FactHTSTestKits ||--o| NDWH_Dim_DimTestKitName : "TestKitName1Key"
    NDWH_Fact_FactHTSTestKits ||--o| NDWH_Dim_DimTestKitName : "TestKitName2Key"
    NDWH_Fact_FactHTSTestKits ||--o| NDWH_Dim_DimTestKitName : "TestKitName3Key"
    NDWH_Fact_FactHTSTestKits ||--o| NDWH_Dim_DimDate : "TestKitExpiry1DateKey"
    NDWH_Fact_FactHTSTestKits ||--o| NDWH_Dim_DimDate : "TestKitExpiry2DateKey"
    NDWH_Fact_FactHTSTestKits ||--o| NDWH_Dim_DimDate : "TestKitExpiry3DateKey"

Loading

Entity Relationship diagram for updated DimFacility table

erDiagram
    NDWH_Dim_DimFacility {
        INT FacilityKey PK
        VARCHAR FacilityName
        VARCHAR Agency
        VARCHAR Ward
        VARCHAR KEPH_Level
        DATE LoadDate
    }

Loading

Entity Relationship diagram for new linelist_hts_tests reporting table

erDiagram
    REPORTING_DATA_EXCHANGE_linelist_hts_tests {
        INT EncounterId
        INT PatientKey
        INT FacilityKey
        INT PartnerKey
        INT AgencyKey
        INT TestKitName1Key
        VARCHAR TestKitLotNumber1
        INT TestKitExpiry1DateKey
        VARCHAR TestResult1
        INT TestKitName2Key
        VARCHAR TestKitLotNumber2
        INT TestKitExpiry2DateKey
        VARCHAR TestResult2
        INT TestKitName3Key
        VARCHAR TestKitLotNumber3
        INT TestKitExpiry3DateKey
        VARCHAR TestResult3
        DATE LoadDate
    }

Loading

Class diagram for updated HTS Test Kit data structures

classDiagram
    class HTS_TestKits {
        +TestKitName1: string
        +TestKitLotNumber1: string
        +TestKitExpiry1: date
        +TestResult1: string
        +TestKitName2: string
        +TestKitLotNumber2: string
        +TestKitExpiry2: date
        +TestResult2: string
        +TestKitName3: string
        +TestKitLotNumber3: string
        +TestKitExpiry3: date
        +TestResult3: string
    }
    class Intermediate_EncounterHTSTests {
        +TestResult1: string
        +TestResult2: string
        +TestResult3: string
        +FinalTestResult: string
        +LoadDate: date
    }
    class FactHTSTestKits {
        +TestKitName1Key: int
        +TestKitLotNumber1: string
        +TestKitExpiry1DateKey: int
        +TestResult1: string
        +TestKitName2Key: int
        +TestKitLotNumber2: string
        +TestKitExpiry2DateKey: int
        +TestResult2: string
        +TestKitName3Key: int
        +TestKitLotNumber3: string
        +TestKitExpiry3DateKey: int
        +TestResult3: string
        +LoadDate: date
    }
    HTS_TestKits <|-- Intermediate_EncounterHTSTests
    Intermediate_EncounterHTSTests <|-- FactHTSTestKits

Loading

Class diagram for updated DimFacility structure

classDiagram
    class DimFacility {
        +FacilityKey: int
        +FacilityName: string
        +Agency: string
        +Ward: string
        +KEPH_Level: string
        +LoadDate: date
    }

Loading

File-Level Changes

Change	Details	Files
Enhanced FactHTSTestKits load to include a third test kit and its expiry dates	Added TestKitName3, TestKitLotNumber3, TestKitExpiry3, TestResult3 to source select Extended target select with TestKitName3Key, TestKitLotNumber3, TestResult3, and three expiry date keys Joined DimTestKitName for the third kit and DimDate for all three expiries Included EncounterId in the output	Scripts/NDWH/HTS_FACT_TABLES/load_FactHTSTestKits.sql
Updated DimTestKitName load to incorporate the third kit name	Added UNION SELECT for TestKitName3 from HTS_TestKits when not null/empty	Scripts/NDWH/ALL DIMENSIONS/load_DimTestKitName.sql
Extended DimFacility load to capture KEPH_Level	Selected KEPH_Level in the source abstraction query Inserted and merged KEPH_Level in target table	Scripts/NDWH/ALL DIMENSIONS/load_DimFacility.sql
Added TestResult3 column to the intermediate HTS encounter tests	Selected TestResult3 alongside existing results Maintained LoadDate consistency	Scripts/ODS/load_intermediate_tables/HTS_intermediate_tables/load_intermediate_EncounterHTSTests.sql
Included TestResult3 in FactHTSClientTests load	Appended TestResult3 to the select list	Scripts/NDWH/HTS_FACT_TABLES/load_FactHTSClientTests.sql
Created linelist_hts_tests in the data-exchange schema with full HTS and kit details	Defined CTEs hts_tests and hts_test_kits pulling in the third kit and expiry date Joined HTS facts, dimension tables, and test kit details Inserted combined fields into DATA_EXCHANGE.dbo.linelist_hts_tests	Scripts/REPORTING/DATA_EXCHANGE/linelist_hts_tests.sql

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey there - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}