PyPI Publishing Blocked by Git Dependencies

[kramaranya]

I've been implementing an automated release process for Kubeflow SDK and when testing it I found one blocking issue -- PyPI publishing fails with this error:

Can't have direct dependency: kubeflow-trainer-api@git+https://github.com/kubeflow/trainer.git@master#subdirectory=api/python_api

Our pyproject.toml contains a git dependency in the dev optional dependencies, that is not allowed when publishing to PyPi:

[project.optional-dependencies]
dev = [
  "pytest>=7.0",
  "pytest-mock>=3.10",
  "coverage>=7.0",
  "kubeflow_trainer_api@git+https://github.com/kubeflow/trainer.git@master#subdirectory=api/python_api",
  "ruff>=0.12.2",
  "pre-commit>=4.2.0",
]

What could we do instead?

1. Remove [project.optional-dependencies].dev entirely and move all dev dependencies to dependency groups:

[dependency-groups]
dev = [
  "pytest>=7.0",
  "pytest-mock>=3.10", 
  "coverage>=7.0",
  "kubeflow-trainer-api @ git+https://github.com/kubeflow/trainer.git@master#subdirectory=api/python_api",
  "ruff>=0.12.2",
  "pre-commit>=4.2.0",
]

This means we don't publish dev extra to PyPi

2. Keep the dev extra but use only published PyPI versions:

[project.optional-dependencies]
dev = [
  "pytest>=7.0",
  "pytest-mock>=3.10",
  "coverage>=7.0", 
  "kubeflow-trainer-api>=2.0.0",  # Published PyPI version only
  "ruff>=0.12.2",
  "pre-commit>=4.2.0",
]

This means our dev extra only tests against stable released versions of kubeflow-trainer-api, not the latest master changes and users can't get Kubeflow SDK with latest changes of trainer API models

3. We can keep dev extra with published versions for PyPI compatibility and add dependency groups for latest changes in kubeflow-trainer-api, that will look smth like that:

[project.optional-dependencies]
dev = [
  "pytest>=7.0",
  "pytest-mock>=3.10",
  "coverage>=7.0",
  "kubeflow-trainer-api>=2.0.0",
  "ruff>=0.12.2", 
  "pre-commit>=4.2.0",
]

[dependency-groups]
dev-latest = [
  {include-group = "dev"},
  "kubeflow-trainer-api @ git+https://github.com/kubeflow/trainer.git@master#subdirectory=api/python_api",
]

This means we have PyPI-compatible dev dependencies and access to latest trainer API changes through dependency groups. However users still can't get latest changes from PyPI

I'd go with option 1, to just use [dependency-groups], since we can test against master trainer API chnages and users who need the latest trainer API changes can install directly from git:

pip install git+https://github.com/kubeflow/sdk.git@main --dependency-groups=dev

Are there any other approaches we can consider? WDYT @andreyvelich @Electronic-Waste @astefanutti

[andreyvelich]

Sure, I am fine to move them to the dependency-groups since we want to allow users to test latest changes as discussed here: #55 (comment)