Python-vscode Dev Template

Adapted from a template I wrote at GESIS for my python development. I've stripped out the internal things (e.g. a local gitlab CICD setup) but some vestiges may remain.

Introduction

This template includes a dev container and vscode configuration for development of dockerised python applications using:

• Python tools:
  • Ruff - formatter
  • Bandit - security checking
  • Poetry - dependency management
  • Pytest - testing
  • Coverage - testing
  • Pyright
  • In precommit:
    • ruff (again)
    • pyupgrade
    • pycln
    • codespell
    • pygrep-hooks
    • check-manifest
    • a bunch of non-python nice to haves
• dev tools:
  • pre-commit

Ideal/Intended Development Flow

This assumes usage of VSCode and optionally development containers (dev containers), though configuration of the desired tools can equally be done on a host machine or a virtual machine.

A less complete version of the template can be used directly on host machines or virtual machines by running make install_locally, and not following the instructions marked [Dev Container]. You may delete the /.devcontainer/ folder if so.

Similarly the template can be used with any code editor but has been configured for VSCode, if you are not using VSCode you may delete the /.vscode/ folder.

Dev containers allow for shared development baselines, to minimise the chance that 'it works on my machine' is uttered during development, as the development container allows for (and should encourage) programmatically defined development infrastructure - and therefore should be the same between developers, and minimise migration issues in deployment or handover.

Installation and Configuration Instructions

1. Configuration of Project
   1. Development environment configuration
      1. Adjust code quality settings in pyproject.toml if desired
         • These are found under [tool.*]
      2. Adjust VSCode development settings in .vscode/settings.json if desired
         • These configure the vscode environment in the project
   2. Project configuration
      1. Change project details in pyproject.toml, Important ones include
         • [project]
         • [tool.flit.module]
      2. Rename ./src/python_package/ to the title of your project
      3. Rename the test import on line 6 of ./tests/test_methods.py to the new name
      4. Edit the ./README.md for your project

Configuring SSH and Git inside the dev container

If one opens the dev container and git diff reports most or all files changed this is likely due to line endings.

If one is unable to pull/push/interact with the remote (due to credentials, so one gets the header when pushing but then is rejected) this is likely because of an outdated local git instance (this is not immediately evident, but sharing the credentials requires git to be fairly recent)

• Ensure the SSH key has been added to the Host Environment
  • (e.g ssh-add -l, ensure not "No identities available")
  • https://code.visualstudio.com/remote/advancedcontainers/sharing-git-credentials

Configuring VSCode

VSCode extensions can be installed manually (as normal) from the host machine if working outside the dev container. If working inside the dev container, rebuilding the container will lose the extensions. You can share make installed extensions persistent inside the dev container, or enable sharing the development environment with colleages, you can change the extensions in /.devcontainer/devcontainer.json inside customizations - vscode - extensions and add the identifier for the extension (found in 'more info', typically to the right hand side). Some optional extensions have been provided in the devcontainer.json.

Credit

• Alex Mühlbauer for your excellent support and guidance
• Microsoft - Daniel Ciborowski
  • Template was adapted from https://github.com/microsoft/python-package-template
• Max Plank Institute
  • https://docs.mpcdf.mpg.de/doc/data/gitlab/devop-tutorial.html

Variables

List of Template Variables and Locations

• ./
  • .devcontainer/
    • devcontainer.json
      • Line 44, pre-commit version, 3.3.2
    • Dockerfile
      • Node version (For Local Sonarlint), 18.x
      • Poetry version, 1.5.1
  • .vscode/
    • settings.json
      • Line length (rulers)
  • readme/
    • N/A
  • src/
    • Dockerfile
      • from python docker container version, latest
      • Poetry version, 1.5.1
      • Line 18: project name py_project_template
      • Line 22: Entrypoint "py_project_template/hello_world.py"
        • May be worth refactoring to a main.py pattern
    • py_project_template/
      • This folder name should be changed
      • __init__.py
        • package version
      • hello_world.py *
  • tests/
    • test_methods.py
      • from py_project_template.hello_world
  • .pre-commit-config.yaml
    • revisions for all repos
  • docker-compose.yml
    • docker compose version
  • pyproject.toml
    • (This should be the authoritative version if possible)
    • Line 2: Poetry version
    • Line 6: Project name
    • Line 7: Project Version
    • Line 8: Authors
    • Line 12: maintainers
    • Line 14: Project Description
    • Line 16: include - project folder
    • Line 19: Minimum python version /
    • Lines 25-39: Versions for:
      • bandit
      • black
      • check-manifest
      • flake8
        • -bugbear
        • -docstrings
        • -formatter_junit_xml
        • -pyproject
      • pylint
      • pytest
        • -cov
        • -mock
        • -runner
      • shellcheck-py
      • coverage
    • Line 45: pyspark versioon
    • Line 54: Line length (black)
    • Line 62: Coverage fail under %
    • Line 65: Line length (flake8)
    • Line 66: Flake include warning types
    • Line 80: Flake ignor warnings
    • Line 98: Python Version
    • Line 99: Python Platform!!!
    • Line 106: TODO: incorporate --cov-fail-under Coverage fail under %
    • Line 186: Line length (pylint)
  • License - LGPL-3.0-or-later

Bump reminders

To bump this to the latest version, update:

• run poetry self update && poetry update --only dev to update poetry and your dependencies.
  • If a new version of poetry, then ensure to update the poetry version number in:
    • pyproject.toml
    • Makefile
    • ./.devcontainer/Dockerfile
• Another manual dependency is pre-commit, update the version number:
  • pre-commit-config.yaml
  • Makefile
  • ./.devcontainer/devcontainer.json
• Run poetry update to update your dependencies.
  • Unfortunately without a script there is no poetry command to upgrade the dependencies in pyproject.toml, for this you can manually iterate through each running poetry add (--group dev/spark) <package>@latest, optionally including the dev and spark groups.
  • N.B. Be careful when updating coverage to include the toml extra field.
• When updating python version:
  • ./.devcontainer/Dockerfile
  • pyproject.toml
    • tool.poetry.dependencies
      • python
    • tool.ruff
      • target-version
    • tool.pyright
      • pythonVersion