PI-5 NGWPC-3694 Delivery #2
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…properly query the HF
warning is expected: Iceberg does not have a dictionary type. <class 'pyarrow.lib.DictionaryType'> will be inferred as int32 on read.
Iceberg4parquets
initial commit of new api prototype
NGWPC-7172 & 7171: Addition of nwm modules ipes
* NGWPC-6953 - Build icefabric endpoint to wrap around the USBR API * NGWPC-6953 - Add Testing and Convert to HTTPX * NGWPC-6953 - Run linting and remove broken RISE result endpoint
* initial commit: created subset CLI function * feat: adding optimized subset function * moved the upstream lookup table logic out of the subset code to add as a pre-defined run * feat: cli tool for building an upstread json file * feat: added build_upstream_connections CLI command * feat: finished an enhancement to the subsetting tool to improve tool performance * added upstream connections json * fix: improved performance in subset_optimized.py * Apply suggestion from @quercoak Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com> * feat: finalized subset work based on conus_hf partitioning * fix: added change in to_geopandas as suggested by @quercoak * tests: added mocks for the hydrofabric catalog in pyiceberg * test: added find_origin tests using mock catalog * tests: finished subset testing locally * patch: all tests are now passing * fixed layers issue in cli * Apply suggestion from @quercoak Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com> * fix: updated code to address PR review * fix: updated sft endpoint and viewer.ipynb notebook to update the subsetting function call * addition of nwm modules ipes * added wgs84 twi cm2m conversion * Update src/icefabric/modules/create_ipes.py Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> * Update src/icefabric/schemas/modules.py Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> * factored in minor changes + update test * Applied pre-commit all passed * NGWPC-6953 - Build icefabric endpoint to wrap around the USBR API (#70) * NGWPC-6953 - Build icefabric endpoint to wrap around the USBR API * NGWPC-6953 - Add Testing and Convert to HTTPX * NGWPC-6953 - Run linting and remove broken RISE result endpoint * tests: added mocks for the hydrofabric catalog in pyiceberg * squashed merge conflicts * tests: removed pyiceberg naming from tests and moved code to hydrofabric/ --------- Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com> Co-authored-by: Sylvia Chin <sylvia.chin@U-1LFS544SOSKLN.nextgenwaterprediction.com> Co-authored-by: SylC-ray <201017958+SylC-ray@users.noreply.github.com> Co-authored-by: Brock Hinkson <brock.w.hinkson@rtx.com>
* initial commit: changed create_ipes to work with mock catalog * fix: changed notebook to comply with new param functions
…#79) * initial commit: changed library to iceberg, created a conus_reference pyiceberg table * feat: added Mike's code from mip_riverML for conflating flowlines to the hydrofabric * fix: added a comment to the confate func * feat: added script to write conflated XS to the icefabric catalog * feat: changed remote_builds to production, added download scripts, edited docs and R code * patch: removed debugging lines * Update tools/iceberg/hydrofabric/download_hydrofabric_gpkg.py Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com> * fix: addressing PR comments --------- Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com>
* NGWPC-7255 - Make NWM Module Endpoints * NGWPC-7255 - Change Hydrofabric mentions to NWM Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> --------- Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com>
* initial commit: added snapshots manager for hydrofabric and schema support * fix: reran hydrofabric parquet file creation * feat: created prod build for hydrofabric data object versioning * NGWPC-7255 - Make NWM Module Endpoints (#78) * NGWPC-7255 - Make NWM Module Endpoints * NGWPC-7255 - Change Hydrofabric mentions to NWM Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> --------- Co-authored-by: Tadd Bindas <Tadd.N.Bindas@rtx.com> * reverting riverML additions * feat: added the update hydrofabric function, bug tested schemas * docs: added a small docs page for the hydrofabric data model * fix: pushing updated mkdocs * patch: updated schema --------- Co-authored-by: Brock Hinkson <brock.w.hinkson@rtx.com>
* patch: added snapshot support and location fix * fix: added logic to update snapshots to the new local versions instead of the glue ones
* feat: adding schema for all representative RAS XS * feat: added code to save all xs from R script * feat: added new schemas to tools * fix: fixed table builds glue
* NGWPC-6905 - Create a CLI for accessing BMI configs * NGWPC-6905 - Fix PR comments/rename nwm_module tags
Addition of NWM modules CFE & UEB ipes
* initial commit: added rustworkx graph support * initial commit: added rustworkx graph support * feat: added support for updating hydrofabric snapshots, added API support * test: added tests for subsets. Will fix catalog mocks later * test: fixed hf subset tests * fix: ensured subsetting will always return a catchment if there is a valid origin * fix:ensured router code removed upstream connections * fix: added ambiance check
This was referenced Sep 30, 2025
|
SWCM witnessed; concurrence granted. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
This is the second PR for NGWPC EDFS deliveries for the icefabric repo and contains a delivery for NGWPC-3694. This work was completed in August of 2025, and new commits to the NGWPC fork have been made since the work went through NGWPC test. Please merge this PR second if merging in all PRs. The full repo commit history is available at: https://github.com/NGWPC/icefabric/commits/main/
To use the remote PyIceberg endpoint, you will need RTX NGWPC AWS creds to our test account. Please contact Tadd Bindas with any questions for access.
Additions
examples/iceberg_data_viewer/ras_xs.ipynbRemovals
Changes
Testing
Screenshots
Notes
Todos
Checklist
Testing checklist
Target Environment support
Accessibility
Other