Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
158 commits
Select commit Hold shift + click to select a range
5699751
Initial commit
taddyb Apr 25, 2025
19ef301
initial commit
taddyb Apr 25, 2025
1fa9f5a
wrote an example for building my warehouse DB
taddyb Apr 26, 2025
08f6dd2
edited build example to add a get_origin() function
taddyb Apr 26, 2025
9db758d
Update LICENSE
taddyb Apr 27, 2025
78db7b8
Update README.md
taddyb Apr 27, 2025
039ace7
updated readme
taddyb Apr 27, 2025
11af641
creating management repo for iceberg catalog
taddyb Apr 28, 2025
6579734
adding new workspaces into the monorepo
taddyb Apr 28, 2025
0d0db9c
updating the example notebook and internal workspace repos so we can …
taddyb Apr 28, 2025
b0abf11
fixed the geopandas function as there was a missing default value for…
taddyb Apr 28, 2025
f462d2f
created the rnr.py file for replace and route hydrofabric functionality
taddyb Apr 29, 2025
67ab989
updated pyproject.toml to reduce python version
taddyb Apr 30, 2025
cb76842
Update pyproject.toml
taddyb Apr 30, 2025
1c28dec
removed dependencies
taddyb Apr 30, 2025
dcda1c4
Update __init__.py
taddyb May 3, 2025
526a804
Update pyproject.toml
taddyb May 4, 2025
c953ccb
Update pyproject.toml
taddyb May 4, 2025
5917c30
lowered numpy versions due to t-route inconsistencies
taddyb May 4, 2025
bdc067d
updated rnr-segments to fix missing flowpath attributes (#1)
taddyb May 6, 2025
e5cbbee
translates parquets with schema into iceberg demo
SylC-ray May 6, 2025
11e99b3
added S3 capability to demo
SylC-ray May 6, 2025
5356ff1
script to transform parquet into apache iceberg
SylC-ray May 6, 2025
a353c20
chore: add boto3 dependency
quercoak May 6, 2025
c285634
chore: uv lock
quercoak May 6, 2025
10d8aca
Update iceberg_table.py
SylC-ray May 6, 2025
79c8640
Merge pull request #2 from NGWPC/iceberg4parquets
SylC-ray May 6, 2025
0c87949
initial commit of new api prototype
danielcumpton May 6, 2025
dfc7b04
Merge pull request #3 from NGWPC/icefabric_api
danielcumpton May 6, 2025
ee06b61
Add demo for xs mapping + updated iceberg demo
SylC-ray May 15, 2025
297265f
updated to accomod namespace & xs & m1_aux data
SylC-ray May 15, 2025
751b0ef
add script mapping XS to IDs categorize by HUC
SylC-ray May 15, 2025
cf1ea69
adding to doc str
SylC-ray May 15, 2025
fde6be6
Add to doc str
SylC-ray May 15, 2025
838481d
Add to doc str
SylC-ray May 15, 2025
04e981e
Update iceberg_table.py
SylC-ray May 15, 2025
50a5535
Update map_fim_data.py
SylC-ray May 15, 2025
a1caada
from icefabric_manage import MapData replacement
SylC-ray May 15, 2025
04e452b
Update __init__.py
SylC-ray May 15, 2025
32e54e8
Merge pull request #4 from NGWPC/xs-m1aux-feature
SylC-ray May 15, 2025
530c15a
removed large file
danielcumpton May 20, 2025
accb584
Update README.md
danielcumpton May 20, 2025
8fe48c2
Merge pull request #6 from NGWPC/module_csv
danielcumpton May 21, 2025
c158200
Xs mip ble feature (#7)
SylC-ray May 23, 2025
d868505
2025.5: Package Updates
quercoak May 23, 2025
62f5f63
2025.5: Pre-commit / linting
quercoak May 23, 2025
1909fd9
Creating the `build/` dir (#13)
taddyb May 27, 2025
facc38c
Observational Streamflow API (#14)
taddyb May 28, 2025
4a82c4d
JIRA-6388: EDFS QA: Create an icefabric tool for exporting icechunk s…
quercoak May 28, 2025
d773b1b
JIRA 6395: Creation of a python version of hfsubsetR (#17)
taddyb May 29, 2025
e9aa780
added api endpoint for the subset geopackage work (#18)
taddyb Jun 2, 2025
544c7cc
NGWPC-6394 - Create LASAM IPE Endpoint (#19)
brock-hinkson Jun 2, 2025
f6c8915
Delete src/icefabric_manage/scripts directory (#22)
taddyb Jun 2, 2025
56b9807
JIRA-6381: EDFS QA: Clean up icefabric_manage repo and examples to cl…
quercoak Jun 3, 2025
b61262b
NGWPC-6384 - Create Icechunk VC Example Notebook
brock-hinkson Jun 3, 2025
dce4ec7
Merge pull request #25 from NGWPC/NGWPC-6384---Create-Icechunk-Versio…
brock-hinkson Jun 4, 2025
d0afb92
NGWPC 6318 + NGWPC 6510: Iceberg Terraform + S3
taddyb Jun 10, 2025
b780858
NGWPC 6399: Write an example Jupyter Notebook showing how to query RA…
taddyb Jun 10, 2025
7876cb4
Docs + API bug cleanup (#29)
taddyb Jun 10, 2025
b6ba4e9
Feature add TIFF conversion to xr.Dataset
SylC-ray Jun 12, 2025
3ba4fcd
Fixed arg doest exist for retrieve_dataset()
SylC-ray Jun 12, 2025
4f1cd34
Demo tif to xr conversion verification
SylC-ray Jun 12, 2025
2df53cc
Adjst made to accom name attribute in arg
SylC-ray Jun 12, 2025
85dc122
Modified import modules
SylC-ray Jun 12, 2025
d01e24b
Add env setup requirement
SylC-ray Jun 16, 2025
c63565b
Update import of icechunk module
SylC-ray Jun 16, 2025
cb01040
Migrates icechunks of TIFFs from local to cloud.
SylC-ray Jun 16, 2025
ba39731
Merge PR NGWPC-6505 from NGWPC/xrdataset_feat_add
SylC-ray Jun 16, 2025
10ee82a
NGWPC-6502: Move the hydrofabric-ui codes into the icefabric repo
quercoak Jun 17, 2025
0381a79
add file to convert all domains to parquet
danielcumpton Jun 17, 2025
f0c65a8
Merge pull request #36 from NGWPC/NGWPC-6509_convert_all_domains_to_p…
danielcumpton Jun 17, 2025
e0db0bb
updates for csv parameters moving to iceberg tables
danielcumpton Jun 26, 2025
64314a9
change csv variable names to iceberg
danielcumpton Jun 26, 2025
ae43749
NGWPC-6591: Complete the topobathy viewer UI workflow
quercoak Jun 27, 2025
1878a49
Chore: Cleaning up uv lock, README, and `tqdm` (#40)
taddyb Jun 27, 2025
e747154
Merge pull request #38 from NGWPC/NGWPC-6398_6538_icefabric_csv_params
danielcumpton Jun 27, 2025
4d263dd
added dotenv() to load creds for API
taddyb Jul 1, 2025
4d86adf
initial commit: added jira story issue template and PR template (#42)
taddyb Jul 2, 2025
54a733f
chore: update PULL_REQUEST_TEMPLATE.md path
taddyb Jul 2, 2025
9f39384
NGWPC 6903: initial commit: refactored the repo based on #41 (#43)
taddyb Jul 7, 2025
40492a6
patch: moving load_creds() into main function of app
taddyb Jul 7, 2025
8540fde
Part I: NGWPC-6893 (#46)
taddyb Jul 8, 2025
f83dba6
Part II: Ngwpc 6893 (#47)
taddyb Jul 9, 2025
9a22d40
Part III: NGWPC-6893 (#48)
taddyb Jul 10, 2025
1702604
NGWPC-6919: Create an API endpoint / method for accessing Albedo data…
quercoak Jul 11, 2025
a6953bb
Remove /v1/modules/parameters route (#53)
quercoak Jul 11, 2025
6ee52fe
Patch: Changing CWD in the SFT example and updating paths (#54)
taddyb Jul 11, 2025
4ed2dae
patch: Update soil_freeze_thaw.ipynb
taddyb Jul 11, 2025
d607f77
Icechunk Tests (#55)
quercoak Jul 14, 2025
f1c27ac
Setting up the OE: Ngwpc 6892 patch (#56)
taddyb Jul 15, 2025
8cbe55f
init factors in ras_xs
SylC-ray Jul 15, 2025
7eed50e
Create __init__.py
SylC-ray Jul 15, 2025
61c2b03
adding filter xs script
SylC-ray Jul 15, 2025
2f6c350
Factor enum for mip, ble, etc
SylC-ray Jul 15, 2025
94a9bd9
adding xs router to main
SylC-ray Jul 15, 2025
a7eebf6
Create __init__.py
SylC-ray Jul 15, 2025
f188102
adding xs router
SylC-ray Jul 15, 2025
89f5e52
Merge pull request #58 from NGWPC/NGWPC-6901v2
SylC-ray Jul 15, 2025
c4fbdca
docs: API update
quercoak Jul 15, 2025
22514c3
xs router test script added
SylC-ray Jul 15, 2025
e5d7099
Add files via upload
SylC-ray Jul 15, 2025
8341cde
fix: albedo test version
Jul 15, 2025
98eb7c6
pre-commit on xs work
Jul 15, 2025
8ae57b0
fix: openapi example
Jul 15, 2025
946279e
Merge pull request #61 from NGWPC/NGWPC-6902
SylC-ray Jul 15, 2025
c0b2446
initial commit: adding SFT documentation to verify support (#63)
taddyb Jul 15, 2025
fa11864
AC Patch: subcase 4-5 (#62)
taddyb Jul 18, 2025
7ca4915
Update pyproject.toml (#67)
taddyb Jul 21, 2025
a62f546
Update pyproject.toml
taddyb Jul 21, 2025
e16658c
patch: fix ras xs notebook glue loading & import (#71)
quercoak Jul 23, 2025
3fe6aa4
addition of nwm modules ipes
Jul 24, 2025
3225018
added wgs84 twi cm2m conversion
Jul 28, 2025
9b8e00f
Update src/icefabric/modules/create_ipes.py
SylC-ray Jul 29, 2025
abb9c97
Update src/icefabric/schemas/modules.py
SylC-ray Jul 29, 2025
88d6f1d
factored in minor changes + update test
Jul 29, 2025
42290ee
Applied pre-commit all passed
Jul 29, 2025
6f10059
Merge pull request #73 from NGWPC/NGWPC-7171
SylC-ray Jul 29, 2025
6537e1f
NGWPC-6953 - Build icefabric endpoint to wrap around the USBR API (#70)
brock-hinkson Jul 29, 2025
59c710b
Ngwpc 6904 (PR attempt 2) (#74)
taddyb Jul 30, 2025
7f15552
Patch: changed create_ipes to work with mock catalog (#75)
taddyb Aug 1, 2025
1764ebe
add UEB & CFE feats
Aug 4, 2025
0aa48bd
update notebook format
Aug 4, 2025
99d6530
PI-5 Subcase 1: Ingesting, Storing, and Categorizing HEC-RAS Models (…
taddyb Aug 6, 2025
e67c94f
update vars nomenclature + urban_frac
Aug 6, 2025
25a73e1
NGWPC-7255 - Make NWM Module Endpoints (#78)
brock-hinkson Aug 6, 2025
3110af3
Ngwpc 7254: Updating the hydrofabric production builds (#80)
taddyb Aug 7, 2025
1a594d5
patch: added snapshot support and location fix (#84)
taddyb Aug 8, 2025
8c68552
feat: adding all representative RAS XS to icefabric (#86)
taddyb Aug 11, 2025
71fc8f3
patch: dropping dups in RAS XS to parquet
taddyb Aug 11, 2025
5785a63
NGWPC-6905 - Create a CLI for accessing BMI configs (#83)
brock-hinkson Aug 12, 2025
5fb5c48
Merge pull request #76 from NGWPC/NGWPC-7253
danielcumpton Aug 12, 2025
25d742b
NGWPC-6940 - Update API to Query Hydrofabric Domain Iceberg History (…
brock-hinkson Aug 13, 2025
34dbfd5
HF API Graph patch (#90)
taddyb Aug 13, 2025
56566cc
fix: including changes missed in #90
taddyb Aug 13, 2025
6c3e18b
patch: fixing missing upstream catchments from subsets introduced in #90
taddyb Aug 14, 2025
c8300f8
initial commit: patched viewer, and added new viewer for ras xs (#92)
taddyb Aug 19, 2025
f4ab8db
PATCH: patched IPE module code to accomidate for new subset code (#94)
taddyb Aug 19, 2025
7eb3347
NGWPC-6956: formatting RISE API outputs for t-route (#93)
taddyb Aug 20, 2025
fee8192
PATCH: changing output to fit 15min interpolation (#95)
taddyb Aug 21, 2025
65f15d2
Update usbr_data_retrieval.py (#96)
taddyb Aug 21, 2025
d921ef6
patch: updating cfs unit conversion (#97)
taddyb Aug 22, 2025
8f97a1b
NGWPC-6073: Add VPU subsetting into icefabric CLI (#100)
quercoak Aug 26, 2025
67db584
adding gage list and GL divide attr R code
danielcumpton Aug 26, 2025
c0b73cf
Merge pull request #103 from NGWPC/NGWPC-6980
danielcumpton Aug 26, 2025
f4f9f9d
NGWPC-6508 - Create CLI endpoint for hourly streamflow (#104)
brock-hinkson Aug 28, 2025
5547528
feat: added pyprojroot to dependencies to solve root path issues (#105)
taddyb Aug 29, 2025
690dc20
initial commit of LSTM notebook
danielcumpton Aug 29, 2025
b092e7d
NGWPC 7006: updating the icefabric rnr subsetting to include latest E…
taddyb Aug 30, 2025
a52d208
commit lstm notebook without outputs
danielcumpton Sep 3, 2025
42c9304
Merge remote-tracking branch 'origin/main' into NGWPC-7481
danielcumpton Sep 3, 2025
cb5e53a
fix comments from PR
danielcumpton Sep 3, 2025
aeea127
Merge pull request #108 from NGWPC/NGWPC-7481
danielcumpton Sep 3, 2025
d8b2a39
Initial GitHub CI workflow and IaC for Test deploy (#109)
kevinspurrier Sep 5, 2025
ca986aa
NGWPC-7483 - Create API Endpoints to Query RAS-XS (#110)
brock-hinkson Sep 9, 2025
b12062e
NGWPC-7484 - Craete example Jupyer Notebook for RAS-XS
brock-hinkson Sep 9, 2025
08a8670
fix: moved example to new location
taddyb Sep 9, 2025
b41d4bb
patch: updated notebook to include API usage
taddyb Sep 10, 2025
3cd186e
merge conflicts
taddyb Sep 30, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/ISSUE_TEMPLATE/capability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
name: JIRA Story/Capability
about: The structure for outlining work being done on a JIRA story
labels: JIRA Story
---

# Capability

<!-- Please link below the JIRA Story for coupling this issue to the scheduled work -->

## Task

<!-- Provide a clear and concise description of the task / work to be done to the repo

* This ticket is meant to do XYZ and to accomplish the following:
- ...
- ...
-->

## Plan/Outline

<!-- Provide a few sentences about how this issue / story will be completed

* This work will be done by creating a BMI module in the XYZ dir
-->


### TODOS

<!-- Create a checklist of what needs to be accomplished in this work -->
- [ ]

### Additional components / Context

<!-- Add any other context or screenshots about the story here. You can also include links to examples of other programs that have something similar to your request. For example:

* Another project [...] solved this by [...]
-->
217 changes: 217 additions & 0 deletions .github/workflows/cicd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
name: CI/CD Pipeline

on:
pull_request:
branches: [main, development]
push:
#TODO: remove dev branch
branches: [main, development]
# Consider how you want to handle version tags
tags: ['v*.*.*']

permissions:
contents: read
packages: write
security-events: write

env:
REGISTRY: ghcr.io
PYTHON_VERSION: '3.13'

jobs:
setup:
runs-on: ubuntu-latest
outputs:
image_base: ${{ steps.vars.outputs.image_base }}
pr_tag: ${{ steps.vars.outputs.pr_tag }}
commit_sha: ${{ steps.vars.outputs.commit_sha }}
commit_sha_short: ${{ steps.vars.outputs.commit_sha_short }}
test_image_tag: ${{ steps.vars.outputs.test_image_tag }}
steps:
- name: Compute image vars
id: vars
shell: bash
run: |
set -euo pipefail
ORG="$(echo "${GITHUB_REPOSITORY_OWNER}" | tr '[:upper:]' '[:lower:]')"
REPO="$(basename "${GITHUB_REPOSITORY}")"
IMAGE_BASE="${REGISTRY}/${ORG}/${REPO}"
echo "image_base=${IMAGE_BASE}" >> "$GITHUB_OUTPUT"

if [ "${GITHUB_EVENT_NAME}" = "pull_request" ]; then
PR_NUM="${{ github.event.pull_request.number }}"
PR_TAG="pr-${PR_NUM}-build"
echo "pr_tag=${PR_TAG}" >> "$GITHUB_OUTPUT"
echo "test_image_tag=${PR_TAG}" >> "$GITHUB_OUTPUT"
fi

if [ "${GITHUB_EVENT_NAME}" = "push" ]; then
COMMIT_SHA="${GITHUB_SHA}"
SHORT_SHA="${COMMIT_SHA:0:12}"
echo "commit_sha=${COMMIT_SHA}" >> "$GITHUB_OUTPUT"
echo "commit_sha_short=${SHORT_SHA}" >> "$GITHUB_OUTPUT"
echo "test_image_tag=${SHORT_SHA}" >> "$GITHUB_OUTPUT"
fi

# TODO: Re-enable this job after linting rules are fixed or the code passes.
# ruff-linting:
# name: Ruff Linting
# runs-on: ubuntu-latest
# steps:
# - name: Checkout code
# uses: actions/checkout@v4
#
# - name: Install uv
# uses: astral-sh/setup-uv@v5
# with:
# enable-cache: true
# cache-dependency-glob: uv.lock pyproject.toml
#
# - name: Set up Python
# uses: astral-sh/setup-uv@v5
# with:
# python-version: "3.13"
# cache-dependency-glob: uv.lock pyproject.toml
#
# - name: Install dependencies
# run: uv sync
#
# - name: Run ruff check
# run: |
# uv run ruff check . --config pyproject.toml --output-format=github
#
# - name: Run ruff formating check
# run: |
# uv run ruff format . --config pyproject.toml --check

# TODO: Re-enable this job after mocking AWS services, or the more integration focused tests are removed.
# The current unit tests require live AWS credentials, which is not practical for a standard CI environment.
# unit-test:
# runs-on: ubuntu-latest
# strategy:
# matrix:
# python-version: ['3.11', '3.12', '3.13']
# steps:
# - name: Checkout code
# uses: actions/checkout@v4
#
# - name: Install uv
# uses: astral-sh/setup-uv@v5
# with:
# enable-cache: true
# python-version: ${{ matrix.python-version }}
# cache-dependency-glob: uv.lock pyproject.toml
#
# - name: Install dependencies
# run: uv sync
#
# - name: Run tests
# run: |
# uv run pytest


build-and-scan:
name: Build and Scan Container
runs-on: ubuntu-latest
needs: setup
steps:
- uses: actions/checkout@v4
- name: Build image for scanning
id: build
uses: docker/build-push-action@v6
with:
context: .
file: ./docker/Dockerfile
# Load the image to the local Docker daemon, but do not push it
load: true
tags: ${{ needs.setup.outputs.image_base }}:${{ needs.setup.outputs.test_image_tag }}
- name: Scan container with Trivy
uses: aquasecurity/trivy-action@0.20.0
with:
# Scan the locally available image
image-ref: ${{ needs.setup.outputs.image_base }}:${{ needs.setup.outputs.test_image_tag }}
format: 'template'
template: '@/contrib/sarif.tpl'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
- name: Upload Trivy SARIF
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: 'trivy-results.sarif'

codeql-scan:
name: CodeQL Scan
if: github.event_name == 'pull_request' || github.event_name == 'push'
runs-on: ubuntu-latest
# The 'unit-test' job is disabled for now. When you re-enable it, uncomment the line below.
# needs: unit-test
permissions:
actions: read
contents: read
security-events: write
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
python-version: ${{ env.PYTHON_VERSION }}
cache-dependency-glob: uv.lock pyproject.toml
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: python
- name: Install dependencies
run: uv sync
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3

publish:
name: Publish to Registry
if: >
github.event_name == 'push' && (
github.ref == 'refs/heads/main' ||
github.ref == 'refs/heads/development' ||
startsWith(github.ref, 'refs/tags/v')
)
runs-on: ubuntu-latest
# When you re-enable your other jobs: ruff-linting, unit-test. Add them to this list.
needs: [setup, build-and-scan, codeql-scan]
steps:
- uses: actions/checkout@v4
- name: Prepare image tags
id: prep_tags
run: |
# Always start with the unique commit SHA tag for traceability
TAGS="${{ needs.setup.outputs.image_base }}:${{ needs.setup.outputs.commit_sha_short }}"

# If it's a push to the main branch, also add the 'latest' tag
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
TAGS="$TAGS,${{ needs.setup.outputs.image_base }}:latest"
fi

# If the trigger was a version tag, add that version as a tag
if [[ "${{ github.ref }}" == refs/tags/v* ]]; then
# github.ref_name holds the tag name (e.g., "v1.0.0")
VERSION_TAG=${{ github.ref_name }}
TAGS="$TAGS,${{ needs.setup.outputs.image_base }}:${VERSION_TAG}"
fi

echo "tags=${TAGS}" >> "$GITHUB_OUTPUT"
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build & push final image
uses: docker/build-push-action@v6
with:
context: .
file: ./docker/Dockerfile
push: true
tags: ${{ steps.prep_tags.outputs.tags }}
Loading