Skip to content

Conversation

@mluck
Copy link
Contributor

@mluck mluck commented Aug 27, 2024

pyogrio seems to have a difficulty writing files when all values in a column are null (None or nan). The workaround here is to use fiona for writing files where pyogrio is explicitly set in geopandas (gpd) by gpd.options.io_engine = "pyogrio". Fixes #1262.

Changes

Adds engine='fiona' to .to_file() in all of the following files

  • data/: esri.py, nld/levee_download.py, usgs/acquire_and_preprocess_3dep_dems.py, usgs/rating_curve_get_usgs_curves.py, wbd/preprocess_wbd.py
  • src/: derive_headwaters.py, derive_level_paths.py, edit_points.py, filter_catchments_and_add_attributes.py, reachID_grid_to_vector_points.py, reachID_grid_to_vector_points.py, split_flows.py, src_adjust_spatial_obs.py, src_roughness_optimization.py, stream_branches.py
  • tools/: eval_plots.py, evaluate_continuity.py, generate_nws_lid.py, make_boxes_from_bounds.py, mosaic_inundation.py, rating_curve_comparison.py, test_case_by_hydro_id.py

Testing

Ran fim_pipeline on HUCs 02030201 and 21010005.

Deployment Plan (For developer use)

How does the changes affect the product?

  • Code only?
  • If applicable, has a deployment plan be created with the deployment person/team?
  • Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
  • If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
  • Require new pre-clip set?
  • Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@mluck mluck added the bug Something isn't working label Aug 27, 2024
EmilyDeardorff
EmilyDeardorff previously approved these changes Sep 6, 2024
Copy link
Contributor

@EmilyDeardorff EmilyDeardorff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this on a FIM pipeline run with the two HUCs where the error was identified and the FIM pipeline ran without issue. Code changes look good, no concerns.

@CarsonPruitt-NOAA CarsonPruitt-NOAA merged commit 5b86929 into dev Sep 13, 2024
1 check passed
@CarsonPruitt-NOAA CarsonPruitt-NOAA deleted the dev-fix-column-error branch September 13, 2024 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[5pt] Pyogrio column errors

3 participants