Skip to content

Conversation

@taddyb
Copy link

@taddyb taddyb commented Sep 30, 2025

Note

This is the first PR for NGWPC EDFS deliveries for the icefabric repo corresponding to PI-4 deliveries. Since work on these deliveries were done in parallel, and this is a new repo, a version release was done with all capabilities included. This work was completed in July of 2025, and new commits have been made since the work went through NGWPC test. Please merge this PR first if merging in all PRs. The full repo commit history is available at: https://github.com/NGWPC/icefabric/commits/main/

To use the remote PyIceberg endpoint, you will need RTX NGWPC AWS creds to our test account. Please contact Tadd Bindas with any questions for access.

Additions

  • Creation of the icefabric repo. An apache iceberg/icechunk implementation of the Hydrofabric and the corresponding data services.

  • NGWPC-3675

    • Update Endpoint for a GET /hydrofabric/modules/parameters Service For the SoilFreezeThaw Module
    • Code for this example is in: examples/initial_parameters/soil_freeze_thaw.ipynb
  • NGWPC-3677

    • Ingest, categorize, store and disseminate new NOS coastal topobathy data
    • Code for this example is in: examples/icechunk_data_viewer/viewer.ipynb
  • NGWPC-3680

    • Update/Create a Hydrofabric data version control system to permit data updates as necessary and to support slowly time varying data.
    • Code for this example is in: examples/iceberg_data_viewer/viewer.ipynb
  • NGWPC-3681

    • Create a UI to present/list cataloged Hydrofabric data, provide manual CRUD methods and indicate what data needs slowly time varying data support.
    • Code for this example is in: examples/iceberg/iceberg_version_control_examples.ipynb, examples/icechunk/icechunk_version_control_examples.ipynb

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Visually tested in supported browsers and devices (see checklist below 👇)
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

  • Windows
  • Linux
  • Browser

Accessibility

  • Keyboard friendly
  • Screen reader friendly

Other

  • Is useable without CSS
  • Is useable without JS
  • Flexible from small to large screens
  • No linting errors or warnings
  • JavaScript tests are passing

taddyb and others added 30 commits April 25, 2025 18:56
warning is expected: Iceberg does not have a dictionary type. <class 'pyarrow.lib.DictionaryType'> will be inferred as int32 on read.
initial commit of new api prototype
quercoak and others added 23 commits July 11, 2025 11:54
* patch: ensured CWD is correct

* updated API image
* initial commit: updated geopackage_viewer to work with iceberg

* feat: updated pyiceberg example to remove s3 reliance

* patch: Update soil_freeze_thaw.ipynb

* Icechunk Tests (#55)

---------

Co-authored-by: Quercus Hamlin <75846376+quercoak@users.noreply.github.com>
* initial commit

* docs: added more docs in the pyiceberg time travel code

* feat: added new notebook example to demonstrate a locally created icechunk repo with virtual references

* patch: added more docs, and example verbiage for PI-4 UAT

* patch: updated timeout settings

* patch: removed /tmp/sft_output/test dir
@taddyb taddyb marked this pull request as ready for review October 1, 2025 19:11
@DJackson2313
Copy link

SWCM witnessed; concurrence granted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

6 participants