Containerization: Relocate path-based dependencies from requirements.txt file to support containerization

[chang-annie]

When rewriteLocalPathDepsInArchives is set to false, the containerization process writes local file paths into the METADATA file inside Poetry generated wheel files that reference monorepo dependencies. During Docker builds, these file paths cannot be resolved, causing build failures. We need to investigate strategies to reliably work around this issue. UV projects do not run into the same issue.

Example METADATA file:

Metadata-Version: 2.3
Name: python_containerization_experiment_lib
Version: 0.1.0.dev0
Summary: Util code for python containerization experiment lib
Requires-Python: >=3.12,<4
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: python-containerization-experiment-util @ file:///Users/anniechang/experiment-python-containerization/util

Example error message:

[INFO] DOCKER> 66.56 Processing /Users/anniechang/experiment-python-containerization/util (from python-containerization-experiment-lib==0.1.0.dev0)
[INFO] DOCKER> 66.56 ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/Users/anniechang/experiment-python-containerization/util'

Steps to Reproduce

1. Create a Poetry project with a Python application module and a docker module to package the application as a Docker image
2. In the docker module, add the containerize-dependencies goal
3. In the application module, set the rewriteLocalPathDepsInArchives to false
4. Build the project and note that the build will fail at the docker module because it cannot resolve the local file path that was included in the generated wheel file
5. (Optional) Unzip the generated wheel file - unzip path/to/generated_wheel.whl and note that there is local file path included in the METADATA file

Requires-Dist: python-containerization-experiment-util @ file:///Users/anniechang/experiment-python-containerization/util

Expected Behavior

Poetry project with monorepo dependencies should be able to successfully containerize when the rewriteLocalPathDepsInArchives is set to `false

Actual Behavior

Poetry project with monorepo dependency fails to containerize because local file path included in the generated wheel file.

Definition of Done:

• Poetry projects should successfully containerize even with path dependencies
  • investigate and implement solution such that final Poetry generated wheel files do not include local file paths when rewriteLocalPathDepsInArchives is set to false
  • leave comment with potential solutions

Test Strategy

• follow steps 1-3 in Steps to Reproduce
• use the containerize-dependencies goal to create a Docker image for the for the docker module
• confirm docker module is able to package the application as a Docker image and containerize successfully

[chang-annie]

DOD completed with @colinpalmer-pro, @epwilkins, and @nartieri

[chang-annie]

Possible solutions:

1. Mount the wheel file at their expected locations (from the METADATA file) in the Docker image
   • Potentially leaks information about filesystem on which the Dockerfile was built
     • But, rewrite local path deps will usually only be set to false during local development, so that mitigates some of the concerns surrounding leaking filesystem information
   • Could also add a step to remove the file path from the final Docker image
2. Mount the wheel file using a variable for the path
   • works but user needs to configure the docker build appropriately, we can't automate that for them
   • potentially brittle because requires use of ${project.parent.parent....}
   • potentially installs dependencies twice
3. Rewrite METADATA on-the-fly within Docker to remove any local file paths
   • This is nice because aligns Poetry’s behavior with UV’s current behavior
   • Wheels also include a RECORD file with hashes so after making changes to METADATA, we need to also regenerate it
   • Potentially brittle solution because need to use regex to remove file paths and maintenance risks
4. Build two versions of the wheel file, one with path dependencies and one without
   • Could potentially add bloat since we’d always be creating 2 wheels
   • Generated wheel files will share the same name so need to use some sort of versioning flag to indicate which is which
     • local-version tag
     • if we do this solution, then generate 2 wheel files for UV too for consistency
   • solution gets complicated because there are many different combinations of whetherrewriteLocalPathDepsInArchives is set & if we're publishing or not
     • we also need to make sure we're publishing the correct wheel file but the current behavior is to rebuild and then grab the most recently created wheel file
5. pip install inside the docker image using --find-links parameter
   • find-links is an array of locations to look for certain packages. You can pass it file paths, individual URLs to TAR or WHEEL files, HTML files, git repositories and more.
   • does not work as standalone solution because the --find-links parameter does not override the embedded file path
   • but can potentially couple this with the solution where we're generating 2 wheel files because --find-links allows us to not have to list out all the wheel files like how we're doing now - https://pip.pypa.io/en/stable/cli/pip_install/#finding-packages
   • based on testing, --find-links does not take precedence over published wheel
     • tested by checking docker build logs to see what version of arbitrary package was being downloaded (url-normalize 2.2.1), then added older version (2.1.0) to our wheelhouse. noted that pip install still picked up 2.2.1 instead of the version in the wheelhouse
6. Use requirements.txt file as source of truth
   • get all path-based dependencies from this file and copy them to wheelhouse and then pip install the wheel files with --no-deps
   • beneficial because it removes the variability of using different resolvers (pip vs. uv vs. poetry)
   • could potentially run into issues with platform-specific builds
7. Revert to original containerization logic of always building from source each time
   • we'd be using sdist to build from source and need to undo the rewriting path dependencies
     • we'd have to change back the sdist pyproject.toml to not include version pinned, just standalone dependencies and then set tool.uv.source section accordingly
     • and do the same for Poetry
     • could add significant docker build time

[chang-annie]

After discussing with @epwilkins and @woods-cpointe, we narrowed down the list of potential solutions to:

1. Mount the wheel file at the expected paths using a variable (e.g., PROJECT_PATH) to define the source location

   • Keep current Docker image build process where the wheel is installed directly with pip
   • But, user must explicitly set PROJECT_PATH. We could provide a default (like ${project.parent.parent...}), but potentially brittle

2. Use requirements.txt file to gather all path-based dependencies, stage them, and update the requirements.txt with new locations.

   • Then, in the Docker image, we can pip install the wheel file with --no-deps setting and install using the requirements.txt as a source of truth (no mismatches due to different resolvers)
   • Requires some guesswork/assumptions to resolve the wheel file names but no more than what we're already assuming (e.g., always in the dist folder, wheel file ending in *-*-none-any.whl format)
   • May run into issues if users have platform-specific wheels but unlikely edge case

3. Revert to previous containerization logic and always build from source distributions (sdist)

   • Will increase build times since each Docker build recompiles from source
   • To make the old logic work, we'd need to "undo" path dependency rewrites that were already done (if rewriteLocalPathDepsInArchives set to true).
   • Expect that there will be more edge cases to come up during implementation

We will move forward with option number 2 of updating the requirements.txt file, which gives us consistent dependency resolution and does not rely on any manual user settings.

[chang-annie]

Feature: Test updating requirements.txt file to rewrite path-based dependencies in preparation to containerize Python applications with monorepo dependencies
  The containerization feature requires some pre-processing of the generated requirements.txt files to remove any path-based dependencies.

  Background:
    Given the following requirement file based at "/<workingdir>/target/my-app/":
      """requirements
      # Direct paths to monorepo projects
      /<workingdir>/target/util
      ../util
      ./nested-util
      # option args for install
      -r other-reqs.txt
      -c constraints.txt
      --index-url=https://my-pypi.org/simple/
      --extra-index-url=my-pvt-index
      #Blank lines should be ignored

      # Editable monorepo projects
      -e /<workingdir>/target/editable-util
      -e ../editable-relative-util
      -e ./editable-nested-util
      # URI-based monorepo projects
      file:///<workingdir>/target/my-app/nested-util
      # named URI-based monorepo projects
      util @ file://localhost/<workingdir>/target/util
      other-util@file:///<workingdir>/target/other-util
      util [extraopt] \
         @ file:///<workingdir>/target/util
      # Package name with pinned version constraint
      util==1.2.3
      # Packages from web
      lib @ https://nightly-build/package/lib
      https://remote-wheelhouse/direct-package.whl
      # Random extra wheel checked into project
      ../extra-packages/leftpad-1.0.0-py3-none-any.whl
      """

  Scenario: Parse through given requirements.txt file and return all path-based requirements
    When path-based requirements are retrieved
    Then the following paths are returned:
      | /<workingdir>/target/util                                          |
      | /<workingdir>/target/other-util                                    |
      | /<workingdir>/target/my-app/nested-util                            |
      | /<workingdir>/target/extra-packages/leftpad-1.0.0-py3-none-any.whl |
      | /<workingdir>/target/editable-util                                 |
      | /<workingdir>/target/editable-relative-util                         |
      | /<workingdir>/target/my-app/editable-nested-util                   |

  Scenario: Rewrites <package-manager> path-based requirements in requirements.txt file
    Given the following path mappings:
      | /<workingdir>/target/util                                          | ${WHEEL_HOUSE}/util-1.0.0-py3-none-any.whl                   |
      | /<workingdir>/target/other-util                                    | ${WHEEL_HOUSE}/other-util-1.0.0-py3-none-any.whl             |
      | /<workingdir>/target/my-app/nested-util                            | ${WHEEL_HOUSE}/nested_util-1.0.0-py3-none-any.whl            |
      | /<workingdir>/target/extra-packages/leftpad-1.0.0-py3-none-any.whl | ${WHEEL_HOUSE}/leftpad-1.0.0-py3-none-any.whl                |
      | /<workingdir>/target/editable-util                                 | ${WHEEL_HOUSE}/editable_util-1.0.0-py3-none-any.whl          |
      | /<workingdir>/target/editable-relative-util                        | ${WHEEL_HOUSE}/editable_relative_util-1.0.0-py3-none-any.whl |
      | /<workingdir>/target/my-app/editable-nested-util                   | ${WHEEL_HOUSE}/editable_nested_util-1.0.0-py3-none-any.whl   |
    When wheels are relocated to staging directory
    Then the requirements file is updated to:
      """requirements
      # Direct paths to monorepo projects
      file://${WHEEL_HOUSE}/util-1.0.0-py3-none-any.whl
      file://${WHEEL_HOUSE}/util-1.0.0-py3-none-any.whl
      file://${WHEEL_HOUSE}/nested_util-1.0.0-py3-none-any.whl
      # option args for install
      -r other-reqs.txt
      -c constraints.txt
      --index-url=https://my-pypi.org/simple/
      --extra-index-url=my-pvt-index
      #Blank lines should be ignored

      # Editable monorepo projects
      file://${WHEEL_HOUSE}/editable_util-1.0.0-py3-none-any.whl
      file://${WHEEL_HOUSE}/editable_relative_util-1.0.0-py3-none-any.whl
      file://${WHEEL_HOUSE}/editable_nested_util-1.0.0-py3-none-any.whl
      # URI-based monorepo projects
      file://${WHEEL_HOUSE}/nested_util-1.0.0-py3-none-any.whl
      # named URI-based monorepo projects
      util @ file://${WHEEL_HOUSE}/util-1.0.0-py3-none-any.whl
      other-util@file://${WHEEL_HOUSE}/other-util-1.0.0-py3-none-any.whl
      util [extraopt] \
         @ file://${WHEEL_HOUSE}/util-1.0.0-py3-none-any.whl
      # Package name with pinned version constraint
      util==1.2.3
      # Packages from web
      lib @ https://nightly-build/package/lib
      https://remote-wheelhouse/direct-package.whl
      # Random extra wheel checked into project
      file://${WHEEL_HOUSE}/leftpad-1.0.0-py3-none-any.whl
      """

[chang-annie]

OTS completed with @lemuelwatts