python-app-template

A template for Python applications.

Features:

• ✅ Standard Python project structure & packaging.
• ✅ Dependency management with pip-tools.
• ✅ Tools for quality checks: documentation, PEP8, typehints, codespell.
• ✅ [Optional] pre-commit hooks to enforce automatic quality checks.
• ✅ Dockerization with focus in image size optimization.
• ✅ Continuous Integration (CI) workflows (GitHub Actions).
• ✅ Continuous Deployment (CI) workflows (Google Cloud Build).
• ✅ Makefile with shortcuts to increase development speed.
• ✅ README badges with project information.
• ✅ Development workflow documentation.
• ✅ Support for Apache Beam integrated pipelines.

Introduction

Write the motivation for this application here. When applicable, include citations for relevant articles or resources for further reading.

The motivation for this repository is to provide a robust, well-documented template for Python applications, including but not limited to GFW data pipelines.

Goals of this template include:

• Reducing the overhead of creating a new Dockerized Python application.
• Minimizing the effort required to transition prototypes to production.
• Establishing consistent development standards across projects.

Usage

Write the relevant documentation on how to use this application here. Ideally, organize it into sections.

Assume in this section that a package has been published in PYPI or a Docker image to the registry.

Use Markdown alerts to highlight important information.

Example:

Caution

Using HTML sections breaks the rendering of Markdown alerts alerts. To avoid this, place them outside <div></div> sections, for example.

Minimum Requirements

This template can be used at various stages of development: Proof of Concept, Prototype, and Production.
Each stage has its own minimum quality requirements and incrementally builds upon the requirements of the previous stages.

1. 💡 Proof of Concept

   • Update the project's name with python init_project.py my-project-name.
   • Declare required dependencies in pyproject.toml. Use the dev extra for development-only dependencies.
   • Store data files (e.g., .txt, .json, .csv) in the src/my_project_name/assets/ directory.
   • Add the following sections to the README.md:
     • Introduction
     • Usage

2. 🛠️ Prototype

   • Set up branch protection rules for main branch (and develop if exists).
     • ☑️ Restrict deletions.
     • ☑️ Require a pull request before merging.
       • ☑️ Require approvals.
       • ☑️ Require conversation resolution before merging.
     • ☑️ Require status checks to pass. Add GitHub actions to checks.
       • ☑️ Require branches to be up to date before merging.
     • ☑️ Block force pushes.
     • ☑️ Allowed merge methods: only Merge.
   • Select a Git Workflow to maintain consistent branching and collaboration practices.
   • Set up Google Cloud Build triggers to automatically publish the Docker image upon merges.
   • Re-compile requirements.txt for Docker using:

     make reqs
     

   • Update the README.md to include:
     • Output Schema

3. 🚀 Production

   • Install and configure pre-commit hooks. See CONTRIBUTING.md.
   • Add unit tests to ensure code reliability.
   • Write thorough documentation:
     • A complete README.md.
     • Docstrings for all public modules, classes, methods and functions.

Repository Overview

This section applies only to the template and provides an overview of the repository contents.

Directories

This is a brief summary of all the relevant directories of the repository.

Directory	Description
.github/	Configuration for GitHub actions.
docs/	Markdown files with detailed documentation.
notebooks/	All jupyter notebooks go here.
src/python_app_template/	All source code go here.
src/python_app_template/assets/	All data files go here.
tests/	All tests go here.

Files

This is a brief summary of all the relevant files of the repository.

File	Description
.dockerignore	Lists files and folders that Docker should ignore when building an image.
.flake8	Configuration for PEP8 checker.
.gitignore	List of files and directories to be ignored by git.
.pre-commit-config.yaml	Configuration to automate software quality checks.
activate-venv.sh	Simple shortcut to enter virtual environment.
cloudbuild.yaml	Configuration to build and publish docker images in Google Cloud.
codecov.yml	Configuration for codecov GitHub integration.
CONTRIBUTING.md	A guide for contributing to the project.
docker-compose.yml	Configuration for docker compose.
Dockerfile	Instructions for building the Docker image.
init_project.py	Automates changing the project name across relevant files and directories.
LICENSE	The software license.
Makefile	Set of commands to simplify development.
MANIFEST.in	Set of patterns to include or exclude files from installed package.
pyproject.toml	Modern Python packaging configuration file.
requirements.txt	Full set of compiled production dependencies (pinned to specific versions).
requirements-test.txt	High level test dependencies needed to test the installed package.
README.md	This file.
setup.py	Legacy Python packaging config file, kept for compatibility with Apache Beam.

Using the CLI

Write instructions on how to use the CLI of the application here.

Config file example

Optional. Provide an example of an input configuration file.

Output schema.

Optional. Discuss any relevant details about the schema. Provide a link to the schema definition.

Data persistence pattern

Optional. Explain the data persistence pattern used in this application.

How to Contribute

Please read the guidelines in CONTRIBUTING.md.

Implementation details

Optional. This section is for describing implementation details, primarily for developers.

Most relevant modules

Optional. Use this section to describe the most important modules of your application.

Example:

Module	Description
cli.py	Defines the application CLI.

Flow chart

Optional. Use this section to display a mermaid diagram that explains the implementation.

References

Optional. Provide any relevant references to support the explanations in this README. Here we provide some examples.

[1] Welch H., Clavelle T., White T. D., Cimino M. A., Van Osdel J., Hochberg T., et al. (2022). Hot spots of unseen fishing vessels. Sci. Adv. 8 (44), eabq2109. doi: 10.1126/sciadv.abq2109

[2] J. H. Ford, B. Bergseth, C. Wilcox, Chasing the fish oil—Do bunker vessels hold the key to fisheries crime networks? Front. Mar. Sci. 5, 267 (2018).