Feat: pems_data package

[thekaveman]

Closes #179

What this PR does

Follows the same general approach and outline from #181 and #184.

New pems_data package

Create a new Python package with its own pyproject.toml listing dependencies and other metadata. The code is organized into the src structure similar to pems_web in #181 and pems_streamlit in #184.

Refactor pems_streamlit data access

#166 originally introduced loading of some key data from S3. This PR refactors that access from out of pems_streamlit and into pems_data in two initial modules:

• pems_data.s3 contains general S3 access classes for PeMS
• pems_data.stations contains station-specific S3 access classes for PeMS

pems_data is introduced as a dependency of pems_streamlit, which then uses the refactored classes in the same was as before.

Devcontainer

Add pems_data to the devcontainer Dockerfile so it is installed (as editable), with its dependencies.

Tests

Ensures coverage is reported for the new pems_data package. No changes needed to CI since pems_data is installed as a dependency of pems_streamlit as normal (we don't need editable installs in CI).

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
pems_data/src/pems_data/services
stations.py
pems_data/src/pems_data/sources
base.py
s3.py
Project Total

This report was generated by python-coverage-comment-action

[lalver1]

This organization is really nice @thekaveman!

There are maybe just a few things that need tweaking since I am testing these changes (started by running bin/build.sh) and I'm seeing this so far:

• web built ✅
• dev built ✅
• streamlit ⚠️
  • ERROR [streamlit app_container 9/9] RUN pip install $(find /wheels -name pems_streamlit*.whl)
  • ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/caltrans/app/pems_data'
  • I think the reason is because streamlit's pyproject lists "pems_data @ file:./pems_data" as a dependency but we have not copied pems_data into the app_container stage. The fix would be to add COPY pems_data pems_data to the streamlit Dockerfile
  • Adding the above gets us over the error and pems_data starts getting installed, but then I see this error
    • Processing ./pems_data (from pems-streamlit==2024.12.2.dev183+gb6191ea.d20250725)
    • Getting requirements to build wheel: finished with status 'error'
    • LookupError: setuptools-scm was unable to detect version for /caltrans/app.
  • The fix is to also copy the .git directory to the app_container stage so that setuptools-scm can see it in the container. This fixes this but then we get this error
    • creating src/pems_data.egg-info
    • error: could not create 'src/pems_data.egg-info': Permission denied
  • I'm testing/troubleshooting this point now but wanted to let you know what I'm seeing so far.

[thekaveman]

@lalver1 yep you are totally right. My bad, I was testing within the devcontainer and didn't run the Streamlit container directly.

Working on a fix...

[thekaveman]

@lalver1 1f9e7d9 should fix this issue, sorry and thanks again for catching it!

Your diagnosis was correct:

I think the reason is because streamlit's pyproject lists "pems_data @ file:./pems_data" as a dependency but we have not copied pems_data into the app_container stage.

And then of course you saw the issues with trying to copy the extra files in...

The issue happens because for the devcontainer, we want the file: dependency, but in the app container we just want to install the package as a package.

This commit makes a few changes. It sort of feels like a "hack", but honestly I wanted to try and avoid having to use a tool like poetry or hatch or something. Those tools were built to solve issues like this, but this workaround also works. LMK what you think:

• Keep the pems_data @ file: ./pems_data reference in pems_streamlit/pyproject.toml -- this is for the devcontainer
• In the Docker build for pems_streamlit:
  • Rewrite the reference to a direct package reference: pems_data @ file: ./pems_data becomes pems_data
  • Build the pems_data and pems_streamlit wheels and download all their dependencies as wheels into a "wheelhouse"
  • Install pems_streamlit from the "wheelhouse"

[thekaveman]

@lalver1 I'm considering whether we maybe want to go the other way...

• Make the reference in pems_streamlit/pyproject.toml just the package pems_data
• In the devcontainer build, rewrite it to the file: ... reference

Maybe that feels cleaner? And also, I'm worried about rewriting a file that is part of the source in terms of calculating the version info, since that looks at git metadata. If we rewrite a file, that would look like a pending change to git, which might change the version calculation.

[thekaveman]

@lalver1 I did go the other way, turns out we don't need to rewrite anything. The direct package dependency reference works in both!

[lalver1]

Nice job @thekaveman! 🙌 It's so nice we didn't need to overwrite anything in the Dockerfile.
The locally running app works fine, and the devcontainer version does too, I can see changes made to the editable packages when running the debugger 👍 I'll just wait for a rebase to approve this.