Skip to content

Python workflows #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 38 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
9b84c17
feat: Add Python testing
GeekMasher Aug 16, 2024
59ed01c
feat: Update testing and add scripts
GeekMasher Aug 16, 2024
a943337
feat: Add initial linting support
GeekMasher Aug 16, 2024
d598602
fix: Update matrix steps
GeekMasher Aug 16, 2024
601181f
fix: Set Matrix issue
GeekMasher Aug 16, 2024
344afd7
fix: Update set matrix script
GeekMasher Aug 16, 2024
14651b5
feat: add needs to jobs
GeekMasher Aug 16, 2024
19c37b7
fix: Update inputs
GeekMasher Aug 16, 2024
d194616
fix: Update typo
GeekMasher Aug 16, 2024
1f49e68
fix: Update output
GeekMasher Aug 16, 2024
5cfc574
feat: Update and in-line again
GeekMasher Aug 16, 2024
c674411
feat: Add general Python workflow
GeekMasher Aug 16, 2024
74ae7c2
feat: Add Patch-Release-Me config
GeekMasher Aug 16, 2024
9e2380d
feat: Add better logging
GeekMasher Aug 16, 2024
161d338
feat: Add better testing support
GeekMasher Aug 16, 2024
47dba7c
feat: Add Python Vendor reusable workflow
GeekMasher Aug 20, 2024
e94f1de
fix: GitHub Token and Env vars
GeekMasher Aug 20, 2024
ba4a936
fix: Custom Property
GeekMasher Aug 20, 2024
d03474e
fix: Add checkout
GeekMasher Aug 20, 2024
a2e3b9e
fix: Output of the custom property
GeekMasher Aug 20, 2024
9375669
feat: Add Update PR support
GeekMasher Aug 20, 2024
46c7100
feat: Add permissions and update PR creation
GeekMasher Aug 20, 2024
7c3bece
fix: Permissions
GeekMasher Aug 20, 2024
f91b7ab
feat: Add change detection support
GeekMasher Aug 20, 2024
32b441a
feat: Update Python workflow to support vendoring
GeekMasher Aug 20, 2024
99ab119
feat: Python Release workflow
GeekMasher Aug 20, 2024
5a87e27
feat: Add GitHub Releases
GeekMasher Aug 20, 2024
b61ce07
fix: Add GH Token
GeekMasher Aug 20, 2024
cbe2591
fix: Update version output
GeekMasher Aug 20, 2024
6801d56
feat: Add GitHub Release
GeekMasher Aug 20, 2024
d1a5bc1
feat: Replace Python release with GitHub release
GeekMasher Aug 20, 2024
b4e19b3
fix: Add check
GeekMasher Aug 20, 2024
c7e3eb4
feat: Add release to Python workflow
GeekMasher Aug 20, 2024
bcae62d
feat: Remove release
GeekMasher Sep 2, 2024
2c52d08
feat: Remove scripts
GeekMasher Sep 2, 2024
027c609
Merge branch 'main' into python-workflows
GeekMasher Sep 3, 2024
b76c822
fix: Update typo
GeekMasher Sep 5, 2024
d9ccea8
fix: Update to main for now
GeekMasher Sep 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions .github/workflows/python-linting.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Python Testing Workflow
#
# - Automatically runs tests on all supported versions of Python
name: Python - Linting

on:
pull_request:
workflow_call:
inputs:
tool:
description: 'The tool to lint with'
type: string
default: 'ruff'
versions:
description: 'Python versions to test against'
type: string
# All Major versions of Python that are currently supported
default: '3.9,3.10,3.11,3.12'

jobs:
python-verions:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: Set matrix
id: set-matrix
run: |
versions="${{ inputs.versions }}"
echo "Version Input :: $versions"
matrix=$(echo "$versions" | tr "," "\n" | awk '{print "\""$1"\""}' | paste -sd "," -)
echo "matrix :: [$matrix]"
echo "matrix=[$matrix]" >> "$GITHUB_OUTPUT"

python-linting:
runs-on: ubuntu-latest
if: ${{ needs.python-verions.outputs.matrix != '[]' }}
needs: [ python-verions ]
strategy:
fail-fast: false
matrix:
python-version: ${{ fromJson(needs.python-verions.outputs.matrix) }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
set -e
if [[ -f pyproject.toml ]]; then
python -m pip install --upgrade pip poetry
poetry install
elif [[ -f Pipfile ]]; then
python -m pip install --upgrade pip pipenv
pipenv sync -d
elif [[ -f requirements.txt ]]; then
python -m pip install --upgrade pip
pip install -r requirements.txt
elif [[ -f Makefile ]]; then
make install
else
echo "No manifest files found to install dependencies"
fi

- name: Run linting
run: |
set -e
TOOL="${{ inputs.tool }}"
if [[ "$TOOL" == "ruff" ]]; then
pip install ruff
ruff check
elif [[ "$TOOL" == "flake8" ]]; then
pip install flake8
flake8 .
elif [[ "$TOOL" == "black" ]]; then
pip install black
black --check .
fi
65 changes: 65 additions & 0 deletions .github/workflows/python-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Python Releasing Workflow
name: Python - Release

on:
push:
workflow_call:
inputs:
version:
description: 'Python main version to vendor'
type: string
# All Major versions of Python that are currently supported
default: '3.11'

permissions:
contents: write
pull-requests: write

jobs:
version-changes:
runs-on: ubuntu-latest
outputs:
release: ${{ steps.check_release.outputs.release }}
version: ${{ steps.check_release.outputs.version }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: "Check release"
id: check_release
env:
GH_TOKEN: ${{ github.token }}
run: |
set -e

if [[ -f .release.yml ]]; then
pip install yq
current_version=$(cat .release.yml | yq -r ".version")
elif [[ -f pyproject.toml ]]; then
current_version=$(grep -oP '^version = "(.*)"$' pyproject.toml | cut -d '"' -f 2)
elif [[ -f setup.py ]]; then
current_version=$(grep -oP '^__version__ = "(.*)"$' setup.py | cut -d '"' -f 2)
else
echo "No version file found"
current_version="NA"
fi

released_version=$(gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" /repos/:owner/:repo/releases/latest | jq -r ".tag_name")

if [[ "$current_version" == "NA" || "$current_version" == "$released_version" ]]; then
echo "No new release found"
echo "release=false" >> "$GITHUB_OUTPUT"
else
echo "New release found"
echo "version=$current_version" >> "$GITHUB_OUTPUT"
echo "release=true" >> "$GITHUB_OUTPUT"
fi

github-release:
uses: advanced-security/reusable-workflows/.github/workflows/release.yml@v0.1.0
needs: [ version-changes ]
if: ${{ needs.version-changes.outputs.release == 'true' }}
secrets: inherit
with:
version: ${{ needs.version-changes.outputs.version }}
85 changes: 85 additions & 0 deletions .github/workflows/python-testing.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Python Testing Workflow
#
# - Automatically runs tests on all supported versions of Python
name: Python - Testing

on:
pull_request:
workflow_call:
inputs:
versions:
description: 'Python versions to test against'
type: string
# All Major versions of Python that are currently supported
default: '3.9,3.10,3.11,3.12'

jobs:
python-verions:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: Set matrix
id: set-matrix
run: |
versions="${{ inputs.versions }}"
echo "Version Input :: $versions"
matrix=$(echo "$versions" | tr "," "\n" | awk '{print "\""$1"\""}' | paste -sd "," -)
echo "matrix :: [$matrix]"
echo "matrix=[$matrix]" >> "$GITHUB_OUTPUT"

python-testing:
# This workflow runs on the latest version of Ubuntu
runs-on: ubuntu-latest
if: ${{ needs.python-verions.outputs.matrix != '[]' }}
needs: [ python-verions ]
strategy:
fail-fast: false
matrix:
python-version: ${{ fromJSON(needs.python-verions.outputs.matrix) }}

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
set -e
echo "Installing dependencies..."
if [[ -f pyproject.toml ]]; then
python -m pip install --upgrade pip poetry
poetry install
elif [[ -f Pipfile ]]; then
python -m pip install --upgrade pip pipenv
pipenv sync -d
elif [[ -f requirements.txt ]]; then
python -m pip install --upgrade pip
pip install -r requirements.txt
elif [[ -f Makefile ]]; then
make install
else
echo "No manifest files found to install dependencies"
fi

- name: Run tests
run: |
set -e
echo "Running Python tests..."
if [[ -f pyproject.toml ]]; then
echo "Running poetry run test"
poetry run test
elif [[ -f Pipfile ]]; then
echo "Running pipenv run test"
pipenv run test
elif [[ -f Makefile ]]; then
echo "Running make test"
make test
else
echo "Unknown test runner..."
echo "Please contact the oss-maintainers team for help."
fi
99 changes: 99 additions & 0 deletions .github/workflows/python-vendor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Python Vendoring Workflow
name: Python - Vendoring

on:
push:
pull_request:
workflow_call:
inputs:
custom-property:
description: 'Name of the custom property to get value from'
type: string
default: 'OSSType'
version:
description: 'Python main version to vendor'
type: string
# All Major versions of Python that are currently supported
default: '3.11'

permissions:
contents: write
pull-requests: write

jobs:
custom-property:
runs-on: ubuntu-latest
outputs:
osstype: ${{ steps.get_custom_property.outputs.osstype }}

steps:
- name: Checkout
uses: actions/checkout@v4
- name: "Get Custom Property"
id: get_custom_property
env:
GH_TOKEN: ${{ github.token }}
PROPERTY_NAME: ${{ inputs.custom-property }}
run: |
set -e

PROPERTIES=$(gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" /repos/:owner/:repo/properties/values)
echo "Properties: '$PROPERTIES'"

REPOSITORY_TYPE=$(echo $PROPERTIES | jq -r ".[] | select(.property_name == \"$PROPERTY_NAME\") | .value")
echo "Repository type: '$REPOSITORY_TYPE'"
echo "osstype=$REPOSITORY_TYPE" >> "$GITHUB_OUTPUT"

python-vendoring:
runs-on: ubuntu-latest
needs: [ custom-property ]
if: ${{ needs.custom-property.outputs.osstype == 'Actions' }}
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Python ${{ inputs.version }}
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.version }}

- name: "Install and Vendor dependencies"
id: vendoring
run: |
set -e

if [[ -f Pipfile ]]; then
python -m pip install --upgrade pip pipenv
pipenv run vendor
elif [[ -f Makefile ]]; then
make vendor
elif [[ -f vendor/update.sh ]]; then
./vendor/update.sh
else
echo "Unknown vendoring method"
fi

CHANGES=$(git status --porcelain | wc -l)
echo "changes=$CHANGES" >> "$GITHUB_OUTPUT"

- name: "Update vendored dependencies (Push)"
if: ${{ steps.vendoring.outputs.changes != 0 && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') }}
uses: peter-evans/create-pull-request@v6
with:
token: ${{ github.token }}
commit-message: "[chore]: Update vendored dependencies"
title: "[chore]: Update vendored dependencies"
branch: update-vendored-dependencies
base: ${{ github.event.before }}
labels: dependencies
body: |
This is an automated PR to update that vendored dependencies are up to date.
It was created by a GitHub workflow defined in `.github/workflows/python-vendor.yml`.
Please do not merge this PR manually.
<details>
<summary>Details</summary>
<p>
This PR was created by a workflow that runs on all pushes to the repository.
It installs dependencies and then verifies that the repository is clean.
</p>
</details>
58 changes: 58 additions & 0 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Python Testing Workflow
#
# - Automatically runs tests on all supported versions of Python
name: Python

on:
pull_request:
workflow_call:
inputs:
version:
description: 'Python main version to vendor'
type: string
# All Major versions of Python that are currently supported
default: '3.11'
versions:
description: 'Python versions to test against'
type: string
# All Major versions of Python that are currently supported
default: '3.9,3.10,3.11,3.12'
vendor:
description: 'Whether to vendor the dependencies'
type: string
default: 'true'


jobs:
# Run the tests on all supported versions of Python
testing:
uses: advanced-security/reusable-workflows/.github/workflows/python-testing.yml@v0.1.0
secrets: inherit
with:
versions: ${{ inputs.versions }}

# Run linters on the codebase
linting:
uses: advanced-security/reusable-workflows/.github/workflows/python-linting.yml@v0.1.0
needs: [ testing ]
secrets: inherit
with:
versions: ${{ inputs.versions }}

# Vendor the dependencies into the repository if needed
vendoring:
uses: advanced-security/reusable-workflows/.github/workflows/python-vendor.yml@v0.1.0
needs: [ testing, linting ]
if: ${{ inputs.vendor == 'true' }}
secrets: inherit
with:
version: ${{ inputs.version }}
custom-property: 'OSSType'

# Release a new version of the package
release:
uses: advanced-security/reusable-workflows/.github/workflows/python-release.yml@v0.1.0
needs: [ testing, linting ]
secrets: inherit
with:
version: ${{ inputs.version }}
Loading