PI-7 NGWPC-5779 delivery #4

taddyb · 2025-09-30T23:27:45Z

Note

This is the fourth PR for NGWPC EDFS deliveries for the icefabric repo and contains a delivery for NGWPC-5779. This work was completed in September of 2025, and new commits to the NGWPC fork have been made since the work went through NGWPC test. Please merge this PR fourth if merging in all PRs. The full repo commit history is available at: https://github.com/NGWPC/icefabric/commits/main/

To use the remote PyIceberg endpoint, you will need RTX NGWPC AWS creds to our test account. Please contact Tadd Bindas with any questions for access.

Additions

NGWPC-5779

  - NWM version 5 activity. 

Collect data as necessary for the LSTM module, adhering to data collection guidelines in 2.4.1: 

- Data collection as necessary. 
- Data extraction and/or value-added information. 
- Services to disseminate the data. 
- Enterprise capabilities (spatial reference, version control, perceptual models, etc.). 

Detail steps include: 
- Extract/Derive value added features of catchment boundaries, land use, land cover, vegetation fraction, soil textures and layers, groundwater basins, aquifer properties, depth to groundwater table, etc., as necessary.
- Build a Data Model in support of these features. 
- Build an accessible platform mediated through a machine-accessible interface which provides multiple representations that are both human and machine readable, providing the 'A' facet of FAIRness. 
- Provide access to metadata, data files, dataset terms, and version information, all of which is indexed and searchable, providing the 'F', 'A', and 'R' facets of FAIRness. 
- Many of these datasets are described using OWP agreed upon ontologies, supporting the 'I' facet of FAIRness. 

Refer also to 2.1.5.12.  (T&M)

Code for this example is in: examples/initial_parameters/lstm.ipynb

Removals

Enhancements to the icefabric code since PI-4 NGWPC: 3675 3677 3680 3681 Deliveries #1 PI-5 NGWPC-3694 Delivery #2 and PI-7 NGWPC-6235 Delivery #3

Changes

Enhancements to the icefabric code since PI-4 NGWPC: 3675 3677 3680 3681 Deliveries #1 PI-5 NGWPC-3694 Delivery #2 and PI-7 NGWPC-6235 Delivery #3

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

Testing checklist

Target Environment support

Windows
Linux
Browser

Accessibility

Keyboard friendly
Screen reader friendly

Other

…properly query the HF

… scans

warning is expected: Iceberg does not have a dictionary type. <class 'pyarrow.lib.DictionaryType'> will be inferred as int32 on read.

Iceberg4parquets

initial commit of new api prototype

…#79) * initial commit: changed library to iceberg, created a conus_reference pyiceberg table * feat: added Mike's code from mip_riverML for conflating flowlines to the hydrofabric * fix: added a comment to the confate func * feat: added script to write conflated XS to the icefabric catalog * feat: changed remote_builds to production, added download scripts, edited docs and R code * patch: removed debugging lines * Update tools/iceberg/hydrofabric/download_hydrofabric_gpkg.py Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com> * fix: addressing PR comments --------- Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com>

* NGWPC-7255 - Make NWM Module Endpoints * NGWPC-7255 - Change Hydrofabric mentions to NWM Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> --------- Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com>

* initial commit: added snapshots manager for hydrofabric and schema support * fix: reran hydrofabric parquet file creation * feat: created prod build for hydrofabric data object versioning * NGWPC-7255 - Make NWM Module Endpoints (#78) * NGWPC-7255 - Make NWM Module Endpoints * NGWPC-7255 - Change Hydrofabric mentions to NWM Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> --------- Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> * reverting riverML additions * feat: added the update hydrofabric function, bug tested schemas * docs: added a small docs page for the hydrofabric data model * fix: pushing updated mkdocs * patch: updated schema --------- Co-authored-by: Brock Hinkson <brock.w.hinkson@rtx.com>

* patch: added snapshot support and location fix * fix: added logic to update snapshots to the new local versions instead of the glue ones

* feat: adding schema for all representative RAS XS * feat: added code to save all xs from R script * feat: added new schemas to tools * fix: fixed table builds glue

* NGWPC-6905 - Create a CLI for accessing BMI configs * NGWPC-6905 - Fix PR comments/rename nwm_module tags

…#89)

* initial commit: added rustworkx graph support * initial commit: added rustworkx graph support * feat: added support for updating hydrofabric snapshots, added API support * test: added tests for subsets. Will fix catalog mocks later * test: fixed hf subset tests * fix: ensured subsetting will always return a catchment if there is a valid origin * fix:ensured router code removed upstream connections * fix: added ambiance check

* initial commit: patched viewer, and added new viewer for ras xs * patch: added glue catalog support for ras_xs

* fix: patched IPE module code to accomidate for new subset code * patch: removed reference to module

* initial commit: moved external rise functions to icefabric source dir * fix: ensured router code removed upstream connections * fix: added print statement for end of tool * initial commit: patched viewer, and added new viewer for ras xs (#92) * initial commit: patched viewer, and added new viewer for ras xs * patch: added glue catalog support for ras_xs * PATCH: patched IPE module code to accomidate for new subset code (#94) * fix: patched IPE module code to accomidate for new subset code * patch: removed reference to module * fix: cleaned up data contained in outputted usbr_data_retrieval.py

* NGWPC-6508 - Create CLI endpoint for hourly streamflow * NGWPC-6508 - Fix CSV generation, add functionality and options * Update tools/icechunk/get_envca_cadwr_txdot_gages_from_s3.py --------- Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com>

…DFS updates (#106) * initial commit: updated rnr to latest icefabric changes * fix: removed debugging endpoints * fix: fixed local error with snapshot table being defined incorrectly

* patch: changed parsing to be outside of the main function for universal access * initial commit of LSTM notebook * commit lstm notebook without outputs * fix comments from PR * patch: changing sample graph to pull from tests/data * patch: remove the gl_hf * NGWPC-7219: Initial IaC and Dependencies Initial take a github action workflow with failing stages commented out to get an image built with mkdocs at the root for IaC work. Terraform IaC required for deploy to Test. --------- Co-authored-by: taddyb <tkb5476@psu.edu> Co-authored-by: Daniel Cumpton <dccumpton@rtx.com>

taddyb · 2025-10-02T14:23:45Z

Cherry picked missing commits from this PR branch. Updating to match OE github history from /efs/edfs-deliveriers/pi_7/subcase_1/icefabric

taddyb and others added 30 commits April 25, 2025 18:56

Initial commit

5699751

initial commit

19ef301

wrote an example for building my warehouse DB

1fa9f5a

edited build example to add a get_origin() function

08f6dd2

Update LICENSE

9db758d

Update README.md

78db7b8

updated readme

039ace7

creating management repo for iceberg catalog

11af641

adding new workspaces into the monorepo

6579734

updating the example notebook and internal workspace repos so we can …

0d0db9c

…properly query the HF

fixed the geopandas function as there was a missing default value for…

b0abf11

… scans

created the rnr.py file for replace and route hydrofabric functionality

f462d2f

updated pyproject.toml to reduce python version

67ab989

Update pyproject.toml

cb76842

removed dependencies

1c28dec

Update __init__.py

dcda1c4

Update pyproject.toml

526a804

Update pyproject.toml

c953ccb

lowered numpy versions due to t-route inconsistencies

5917c30

updated rnr-segments to fix missing flowpath attributes (#1)

bdc067d

translates parquets with schema into iceberg demo

e5cbbee

added S3 capability to demo

11e99b3

script to transform parquet into apache iceberg

5356ff1

chore: add boto3 dependency

a353c20

chore: uv lock

c285634

Update iceberg_table.py

10d8aca

warning is expected: Iceberg does not have a dictionary type. <class 'pyarrow.lib.DictionaryType'> will be inferred as int32 on read.

Merge pull request #2 from NGWPC/iceberg4parquets

79c8640

Iceberg4parquets

initial commit of new api prototype

0c87949

Merge pull request #3 from NGWPC/icefabric_api

dfc7b04

initial commit of new api prototype

Add demo for xs mapping + updated iceberg demo

ee06b61

Sylvia Chin and others added 28 commits October 2, 2025 10:19

update notebook format

5a8fe1f

update vars nomenclature + urban_frac

576fc9f

NGWPC-7255 - Make NWM Module Endpoints (#78)

1b2a6fe

* NGWPC-7255 - Make NWM Module Endpoints * NGWPC-7255 - Change Hydrofabric mentions to NWM Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> --------- Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com>

patch: added snapshot support and location fix (#84)

791c073

* patch: added snapshot support and location fix * fix: added logic to update snapshots to the new local versions instead of the glue ones

feat: adding all representative RAS XS to icefabric (#86)

ec9e7ce

* feat: adding schema for all representative RAS XS * feat: added code to save all xs from R script * feat: added new schemas to tools * fix: fixed table builds glue

patch: dropping dups in RAS XS to parquet

3aa4dd7

NGWPC-6905 - Create a CLI for accessing BMI configs (#83)

2e4d72d

* NGWPC-6905 - Create a CLI for accessing BMI configs * NGWPC-6905 - Fix PR comments/rename nwm_module tags

NGWPC-6940 - Update API to Query Hydrofabric Domain Iceberg History (…

e336803

…#89)

fix: including changes missed in #90

5f3e3ae

patch: fixing missing upstream catchments from subsets introduced in #90

3097e7c

initial commit: patched viewer, and added new viewer for ras xs (#92)

a1addc7

* initial commit: patched viewer, and added new viewer for ras xs * patch: added glue catalog support for ras_xs

PATCH: patched IPE module code to accomidate for new subset code (#94)

b741512

* fix: patched IPE module code to accomidate for new subset code * patch: removed reference to module

PATCH: changing output to fit 15min interpolation (#95)

198e242

Update usbr_data_retrieval.py (#96)

650b8d5

patch: updating cfs unit conversion (#97)

d1f6a50

NGWPC-6073: Add VPU subsetting into icefabric CLI (#100)

65ea73c

adding gage list and GL divide attr R code

d7f2d7c

feat: added pyprojroot to dependencies to solve root path issues (#105)

05377d0

initial commit of LSTM notebook

74a0baf

NGWPC 7006: updating the icefabric rnr subsetting to include latest E…

535e480

…DFS updates (#106) * initial commit: updated rnr to latest icefabric changes * fix: removed debugging endpoints * fix: fixed local error with snapshot table being defined incorrectly

commit lstm notebook without outputs

b00c59b

fix comments from PR

a941c60

CarsonPruitt-NOAA added this to NGWPC Oct 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PI-7 NGWPC-5779 delivery #4

PI-7 NGWPC-5779 delivery #4

Uh oh!

taddyb commented Sep 30, 2025

Uh oh!

taddyb commented Oct 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

PI-7 NGWPC-5779 delivery #4

Are you sure you want to change the base?

PI-7 NGWPC-5779 delivery #4

Uh oh!

Conversation

taddyb commented Sep 30, 2025

Additions

Removals

Changes

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

Testing checklist

Target Environment support

Accessibility

Other

Uh oh!

taddyb commented Oct 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants