Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
5699751
Initial commit
taddyb Apr 25, 2025
19ef301
initial commit
taddyb Apr 25, 2025
1fa9f5a
wrote an example for building my warehouse DB
taddyb Apr 26, 2025
08f6dd2
edited build example to add a get_origin() function
taddyb Apr 26, 2025
9db758d
Update LICENSE
taddyb Apr 27, 2025
78db7b8
Update README.md
taddyb Apr 27, 2025
039ace7
updated readme
taddyb Apr 27, 2025
11af641
creating management repo for iceberg catalog
taddyb Apr 28, 2025
6579734
adding new workspaces into the monorepo
taddyb Apr 28, 2025
0d0db9c
updating the example notebook and internal workspace repos so we can …
taddyb Apr 28, 2025
b0abf11
fixed the geopandas function as there was a missing default value for…
taddyb Apr 28, 2025
f462d2f
created the rnr.py file for replace and route hydrofabric functionality
taddyb Apr 29, 2025
67ab989
updated pyproject.toml to reduce python version
taddyb Apr 30, 2025
cb76842
Update pyproject.toml
taddyb Apr 30, 2025
1c28dec
removed dependencies
taddyb Apr 30, 2025
dcda1c4
Update __init__.py
taddyb May 3, 2025
526a804
Update pyproject.toml
taddyb May 4, 2025
c953ccb
Update pyproject.toml
taddyb May 4, 2025
5917c30
lowered numpy versions due to t-route inconsistencies
taddyb May 4, 2025
bdc067d
updated rnr-segments to fix missing flowpath attributes (#1)
taddyb May 6, 2025
e5cbbee
translates parquets with schema into iceberg demo
SylC-ray May 6, 2025
11e99b3
added S3 capability to demo
SylC-ray May 6, 2025
5356ff1
script to transform parquet into apache iceberg
SylC-ray May 6, 2025
a353c20
chore: add boto3 dependency
quercoak May 6, 2025
c285634
chore: uv lock
quercoak May 6, 2025
10d8aca
Update iceberg_table.py
SylC-ray May 6, 2025
79c8640
Merge pull request #2 from NGWPC/iceberg4parquets
SylC-ray May 6, 2025
0c87949
initial commit of new api prototype
danielcumpton May 6, 2025
dfc7b04
Merge pull request #3 from NGWPC/icefabric_api
danielcumpton May 6, 2025
ee06b61
Add demo for xs mapping + updated iceberg demo
SylC-ray May 15, 2025
297265f
updated to accomod namespace & xs & m1_aux data
SylC-ray May 15, 2025
751b0ef
add script mapping XS to IDs categorize by HUC
SylC-ray May 15, 2025
cf1ea69
adding to doc str
SylC-ray May 15, 2025
fde6be6
Add to doc str
SylC-ray May 15, 2025
838481d
Add to doc str
SylC-ray May 15, 2025
04e981e
Update iceberg_table.py
SylC-ray May 15, 2025
50a5535
Update map_fim_data.py
SylC-ray May 15, 2025
a1caada
from icefabric_manage import MapData replacement
SylC-ray May 15, 2025
04e452b
Update __init__.py
SylC-ray May 15, 2025
32e54e8
Merge pull request #4 from NGWPC/xs-m1aux-feature
SylC-ray May 15, 2025
530c15a
removed large file
danielcumpton May 20, 2025
accb584
Update README.md
danielcumpton May 20, 2025
8fe48c2
Merge pull request #6 from NGWPC/module_csv
danielcumpton May 21, 2025
c158200
Xs mip ble feature (#7)
SylC-ray May 23, 2025
d868505
2025.5: Package Updates
quercoak May 23, 2025
62f5f63
2025.5: Pre-commit / linting
quercoak May 23, 2025
1909fd9
Creating the `build/` dir (#13)
taddyb May 27, 2025
facc38c
Observational Streamflow API (#14)
taddyb May 28, 2025
4a82c4d
JIRA-6388: EDFS QA: Create an icefabric tool for exporting icechunk s…
quercoak May 28, 2025
d773b1b
JIRA 6395: Creation of a python version of hfsubsetR (#17)
taddyb May 29, 2025
e9aa780
added api endpoint for the subset geopackage work (#18)
taddyb Jun 2, 2025
544c7cc
NGWPC-6394 - Create LASAM IPE Endpoint (#19)
brock-hinkson Jun 2, 2025
f6c8915
Delete src/icefabric_manage/scripts directory (#22)
taddyb Jun 2, 2025
56b9807
JIRA-6381: EDFS QA: Clean up icefabric_manage repo and examples to cl…
quercoak Jun 3, 2025
b61262b
NGWPC-6384 - Create Icechunk VC Example Notebook
brock-hinkson Jun 3, 2025
dce4ec7
Merge pull request #25 from NGWPC/NGWPC-6384---Create-Icechunk-Versio…
brock-hinkson Jun 4, 2025
d0afb92
NGWPC 6318 + NGWPC 6510: Iceberg Terraform + S3
taddyb Jun 10, 2025
b780858
NGWPC 6399: Write an example Jupyter Notebook showing how to query RA…
taddyb Jun 10, 2025
7876cb4
Docs + API bug cleanup (#29)
taddyb Jun 10, 2025
b6ba4e9
Feature add TIFF conversion to xr.Dataset
SylC-ray Jun 12, 2025
3ba4fcd
Fixed arg doest exist for retrieve_dataset()
SylC-ray Jun 12, 2025
4f1cd34
Demo tif to xr conversion verification
SylC-ray Jun 12, 2025
2df53cc
Adjst made to accom name attribute in arg
SylC-ray Jun 12, 2025
85dc122
Modified import modules
SylC-ray Jun 12, 2025
d01e24b
Add env setup requirement
SylC-ray Jun 16, 2025
c63565b
Update import of icechunk module
SylC-ray Jun 16, 2025
cb01040
Migrates icechunks of TIFFs from local to cloud.
SylC-ray Jun 16, 2025
ba39731
Merge PR NGWPC-6505 from NGWPC/xrdataset_feat_add
SylC-ray Jun 16, 2025
10ee82a
NGWPC-6502: Move the hydrofabric-ui codes into the icefabric repo
quercoak Jun 17, 2025
0381a79
add file to convert all domains to parquet
danielcumpton Jun 17, 2025
f0c65a8
Merge pull request #36 from NGWPC/NGWPC-6509_convert_all_domains_to_p…
danielcumpton Jun 17, 2025
e0db0bb
updates for csv parameters moving to iceberg tables
danielcumpton Jun 26, 2025
64314a9
change csv variable names to iceberg
danielcumpton Jun 26, 2025
ae43749
NGWPC-6591: Complete the topobathy viewer UI workflow
quercoak Jun 27, 2025
1878a49
Chore: Cleaning up uv lock, README, and `tqdm` (#40)
taddyb Jun 27, 2025
e747154
Merge pull request #38 from NGWPC/NGWPC-6398_6538_icefabric_csv_params
danielcumpton Jun 27, 2025
4d263dd
added dotenv() to load creds for API
taddyb Jul 1, 2025
4d86adf
initial commit: added jira story issue template and PR template (#42)
taddyb Jul 2, 2025
54a733f
chore: update PULL_REQUEST_TEMPLATE.md path
taddyb Jul 2, 2025
9f39384
NGWPC 6903: initial commit: refactored the repo based on #41 (#43)
taddyb Jul 7, 2025
40492a6
patch: moving load_creds() into main function of app
taddyb Jul 7, 2025
8540fde
Part I: NGWPC-6893 (#46)
taddyb Jul 8, 2025
f83dba6
Part II: Ngwpc 6893 (#47)
taddyb Jul 9, 2025
9a22d40
Part III: NGWPC-6893 (#48)
taddyb Jul 10, 2025
1702604
NGWPC-6919: Create an API endpoint / method for accessing Albedo data…
quercoak Jul 11, 2025
a6953bb
Remove /v1/modules/parameters route (#53)
quercoak Jul 11, 2025
6ee52fe
Patch: Changing CWD in the SFT example and updating paths (#54)
taddyb Jul 11, 2025
4ed2dae
patch: Update soil_freeze_thaw.ipynb
taddyb Jul 11, 2025
d607f77
Icechunk Tests (#55)
quercoak Jul 14, 2025
f1c27ac
Setting up the OE: Ngwpc 6892 patch (#56)
taddyb Jul 15, 2025
8cbe55f
init factors in ras_xs
SylC-ray Jul 15, 2025
7eed50e
Create __init__.py
SylC-ray Jul 15, 2025
61c2b03
adding filter xs script
SylC-ray Jul 15, 2025
2f6c350
Factor enum for mip, ble, etc
SylC-ray Jul 15, 2025
94a9bd9
adding xs router to main
SylC-ray Jul 15, 2025
a7eebf6
Create __init__.py
SylC-ray Jul 15, 2025
f188102
adding xs router
SylC-ray Jul 15, 2025
89f5e52
Merge pull request #58 from NGWPC/NGWPC-6901v2
SylC-ray Jul 15, 2025
c4fbdca
docs: API update
quercoak Jul 15, 2025
22514c3
xs router test script added
SylC-ray Jul 15, 2025
e5d7099
Add files via upload
SylC-ray Jul 15, 2025
8341cde
fix: albedo test version
Jul 15, 2025
98eb7c6
pre-commit on xs work
Jul 15, 2025
8ae57b0
fix: openapi example
Jul 15, 2025
946279e
Merge pull request #61 from NGWPC/NGWPC-6902
SylC-ray Jul 15, 2025
c0b2446
initial commit: adding SFT documentation to verify support (#63)
taddyb Jul 15, 2025
fa11864
AC Patch: subcase 4-5 (#62)
taddyb Jul 18, 2025
7ca4915
Update pyproject.toml (#67)
taddyb Jul 21, 2025
a62f546
Update pyproject.toml
taddyb Jul 21, 2025
e16658c
patch: fix ras xs notebook glue loading & import (#71)
quercoak Jul 23, 2025
3fe6aa4
addition of nwm modules ipes
Jul 24, 2025
3225018
added wgs84 twi cm2m conversion
Jul 28, 2025
9b8e00f
Update src/icefabric/modules/create_ipes.py
SylC-ray Jul 29, 2025
abb9c97
Update src/icefabric/schemas/modules.py
SylC-ray Jul 29, 2025
88d6f1d
factored in minor changes + update test
Jul 29, 2025
42290ee
Applied pre-commit all passed
Jul 29, 2025
6f10059
Merge pull request #73 from NGWPC/NGWPC-7171
SylC-ray Jul 29, 2025
6537e1f
NGWPC-6953 - Build icefabric endpoint to wrap around the USBR API (#70)
brock-hinkson Jul 29, 2025
59c710b
Ngwpc 6904 (PR attempt 2) (#74)
taddyb Jul 30, 2025
7f15552
Patch: changed create_ipes to work with mock catalog (#75)
taddyb Aug 1, 2025
1764ebe
add UEB & CFE feats
Aug 4, 2025
0aa48bd
update notebook format
Aug 4, 2025
99d6530
PI-5 Subcase 1: Ingesting, Storing, and Categorizing HEC-RAS Models (…
taddyb Aug 6, 2025
e67c94f
update vars nomenclature + urban_frac
Aug 6, 2025
25a73e1
NGWPC-7255 - Make NWM Module Endpoints (#78)
brock-hinkson Aug 6, 2025
3110af3
Ngwpc 7254: Updating the hydrofabric production builds (#80)
taddyb Aug 7, 2025
1a594d5
patch: added snapshot support and location fix (#84)
taddyb Aug 8, 2025
8c68552
feat: adding all representative RAS XS to icefabric (#86)
taddyb Aug 11, 2025
71fc8f3
patch: dropping dups in RAS XS to parquet
taddyb Aug 11, 2025
5785a63
NGWPC-6905 - Create a CLI for accessing BMI configs (#83)
brock-hinkson Aug 12, 2025
5fb5c48
Merge pull request #76 from NGWPC/NGWPC-7253
danielcumpton Aug 12, 2025
25d742b
NGWPC-6940 - Update API to Query Hydrofabric Domain Iceberg History (…
brock-hinkson Aug 13, 2025
34dbfd5
HF API Graph patch (#90)
taddyb Aug 13, 2025
56566cc
fix: including changes missed in #90
taddyb Aug 13, 2025
6c3e18b
patch: fixing missing upstream catchments from subsets introduced in #90
taddyb Aug 14, 2025
518f7a8
initial commit: patched viewer, and added new viewer for ras xs
taddyb Aug 14, 2025
909514d
patch: added glue catalog support for ras_xs
taddyb Aug 15, 2025
03c28dc
merge conflicts
taddyb Sep 30, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/ISSUE_TEMPLATE/capability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
name: JIRA Story/Capability
about: The structure for outlining work being done on a JIRA story
labels: JIRA Story
---

# Capability

<!-- Please link below the JIRA Story for coupling this issue to the scheduled work -->

## Task

<!-- Provide a clear and concise description of the task / work to be done to the repo

* This ticket is meant to do XYZ and to accomplish the following:
- ...
- ...
-->

## Plan/Outline

<!-- Provide a few sentences about how this issue / story will be completed

* This work will be done by creating a BMI module in the XYZ dir
-->


### TODOS

<!-- Create a checklist of what needs to be accomplished in this work -->
- [ ]

### Additional components / Context

<!-- Add any other context or screenshots about the story here. You can also include links to examples of other programs that have something similar to your request. For example:

* Another project [...] solved this by [...]
-->
265 changes: 202 additions & 63 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,77 +1,216 @@
# Compiled source #
###################
*.com
*.class
*.dll
*.exe
*.o
*.so
_site/

# Packages #
############
# it's better to unpack these files and commit the raw source
# git has its own built in compression methods
*.7z
*.dmg
*.gz
*.iso
*.jar
*.rar
*.tar
*.zip

# Logs and databases #
######################
*.log
*.sql
*.sqlite

# OS generated files #
######################
.DS_Store
.DS_Store?
.Spotlight-V100
.Trashes
Icon?
ehthumbs.db
Thumbs.db

# Vim swap files #
##################
*.swp

# Python #
#################
*.pyc
*.egg-info/
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
.env
.python-version
*$py.class

# pyenv #
#########
.python-version
# C extensions
*.so

# Django #
#################
*.egg-info
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
.DS_Store

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
#################
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Front-End #
#############
node_modules/
bower_components/
.grunt/
src/vendor/
dist/
# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
uv.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
.vscode/

# Ruff stuff:
.ruff_cache/

# PyPI configuration file
.pypirc

Pipfile

.qodo
src/icefabric/_version.py
*.db
*.zarr
.zarr
*.tiff
*.tif
examples/iceberg_catalog/metadata/

# Local Terraform state
*.tfstate
*.tfstate.backup

# Crash log files
crash.log

# CLI configuration
.terraform/

# Terraform plan output (optional unless you want to inspect later)
*.tfplan

# tiles
*.pmtiles
examples/icechunk_data_viewer/martin/tiles/quantiles/*
examples/icechunk_data_viewer/martin/tiles/legends/*
examples/icechunk_data_viewer/martin/tiles/legend.png
tests/data/topo_tifs

# all data files
data

# R code
.Rproj.user
.Rdata
.Rhistory
icefabric.Rproj
31 changes: 31 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
exclude: LICENSE|\.csv$
- id: end-of-file-fixer
exclude: LICENSE|\.csv$
- id: check-yaml
exclude: mkdocs.yml$
- id: debug-statements

- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.12.3
hooks:
# Run the linter.
- id: ruff
args: [ --fix ]
# Run the formatter.
- id: ruff-format

- repo: https://github.com/astral-sh/uv-pre-commit
rev: 0.7.20
hooks:
- id: uv-lock

- repo: https://github.com/kynan/nbstripout
rev: 0.8.1
hooks:
- id: nbstripout
11 changes: 11 additions & 0 deletions .pyiceberg.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
catalog:
glue:
type: glue
s3.endpoint: s3.us-east-1.amazonaws.com
warehouse: s3://52fcde3e7-5582-477d-7686ou4ij1ptxj8equ83a5xc51fsuse1b--table-s3
region: us-east-1
glue_region: us-east-1
sql:
type: sql
uri: sqlite:////tmp/warehouse/pyiceberg_catalog.db
warehouse: file:///tmp/warehouse
11 changes: 11 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
Copyright 2025 Raytheon Company

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

Licensed under: https://opensource.org/license/bsd-2-clause

- - - - - - - - - - - - - -


Apache License
Version 2.0, January 2004
Expand Down
14 changes: 14 additions & 0 deletions NOTICE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
--

The Hydrofabric data used in this repo is credited to Lynker and its contributors

Johnson, J. M. (2022). National Hydrologic Geospatial Fabric (hydrofabric) for the Next Generation (NextGen) Hydrologic Modeling Framework,
HydroShare http://www.hydroshare.org/resource/129787b468aa4d55ace7b124ed27dbde

--

Inspiration and assistance in the creation, and storage of network graphs comes from the following repos. Credit to the authors:
- https://github.com/DeepGroundwater/ddr/blob/master/engine/adjacency.py
- https://github.com/CIROH-UA/NGIAB_data_preprocess/blob/main/modules/data_processing/graph_utils.py

--
Loading