Skip to content

Conversation

ZahraGhahremani
Copy link
Contributor

@ZahraGhahremani ZahraGhahremani commented Sep 11, 2025

Adjusted the script to source columns from src_base instead of src_full .

Notes Regarding Ghost Bug

The src_base files are generated directly by TauDEM, not by our post-processing scripts.
The slope values in src_base represent TauDEM’s rise-over-run slopes, written in its standard double-precision format.

Because these slope values—and the subsequent slopes propagated through HFAB and SWORD—are extremely small (e.g., 9.99999974737875E-06), it is essential to preserve their numerical precision across all read/write operations in downstream scripts. Even minor floating-point rounding or formatting differences can lead to inconsistencies in later hydraulic computations.


Adopted Approach

When writing slope values to derived files (e.g., src_full, hydrotables):

  • Multiply each slope by 1e9
  • Round to zero decimal places
  • Store the result as an integer

Example: 9.99999974737875E-06 → 10000

Maintain this integer fixed-point representation in all subsequent files and processes.

Whenever slope values are required for computation (e.g., in Manning’s equation to compute discharge), divide the stored integers by 1e9 to recover the original floating-point slope, e.g., 10000 / 1e9 = 1e-5.

This method ensures consistent results across all post-processing stages.

Changes

dev-fix-update-hydrotable: as described.


Testing

Generally, you do not copy this part into the ChangeLog. These are some quick notes on what you did test and/or notes for the reviewer to help with their review testing.


Deployment Plan (For FIM developers use)

  • Does the change impact inputs, docker or python packages?

    • Yes
    • No (f no.. skip the rest of the Deployment Plan section)
  • If you are not a FIM dev team member: Please let us know what you need and we can help with it.

  • If you are a FIM Dev team member:

    • Please work with the DevOps team and do not just go ahead and do it without some co-ordination.

    • Copy where you can, assign where you can not, and it is your responsibility to ensure it is done. Please ensure it is completed before the PR is merged.

    • Has new or updated python packages, PipFile, Pipefile.lock or Dockerfile changes? DevOps can help or take care of it if you want. Just need to know if it is required.

      • Yes
      • No
    • Require new or adjusted data inputs? Does it have a way to version (folder or file dates)?

      • No
      • Yes
        • Require new pre-clip set or any other data reloads, such as DEMS, osm, etc. ie.. pre-requisite re-data upstream of your input changes.
          • Yes
          • No
        • Has the inputs been copied/exist in all five enviros:
          • FIM EFS
          • FIM S3
          • ESIP
          • Dev1
          • UCS2
  • Please use caution in removing older version unless it is at least two versions ago. Confirm with DevOps if cleanup might be involved.

  • If new or updated data sets, has the FIM code, including running fim_pipeline.sh, been updated and tested with the new/adjusted data? You can dev test against subsets if you like.

    • Yes

Notes to DevOps Team or others:

Please add any notes that are helpful for us to make sure it is all done correctly. Do not put actual server names or full true paths, just shortcut paths like 'efs..../inputs/, or 'dev1....inputs', etc.


Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@ZahraGhahremani ZahraGhahremani added the bug Something isn't working label Sep 11, 2025
@ZahraGhahremani ZahraGhahremani marked this pull request as ready for review September 11, 2025 23:48
@AliForghani-NOAA AliForghani-NOAA self-requested a review October 9, 2025 15:43
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this and this line because we already did that here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RyanSpies-NOAA and @hhs732 Can you confirm that slope is not being changed during any of the post-processing scripts. If it is changing during post-processing, this may be problematic?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we replace this block with just below two lines:

recalc_df =input_src_base.copy()
recalc_df = recalc_df.merge(input_src_full[['HydroID', 'Stage', 'SLOPE']], on=['HydroID', 'Stage'], how='left' )

If needed, you can then drop any unwanted columns carried over from input_src_base.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my understanding of the workflow to reset src_full and hydrotable using this code:

  • make an empty recalc_df:

    • get slope and manning from current src_full
    • get the rest from src_base (number of cells,...)
    • calculate discharge
  • update src_full with recalc_df values (discharge and all other fields)

  • update discharge_cms and default_discharge_cms columns of hydro_table with recalc_df discharge column.

@ZahraGhahremani If this is correct, please include it in the code’s docstring.

@AliForghani-NOAA AliForghani-NOAA self-assigned this Oct 22, 2025
@AliForghani-NOAA AliForghani-NOAA removed the request for review from hhs732 October 23, 2025 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants