poiesis

TES (Task Execution Service) on kubernetes!!!

A TES compliant task execution service built on kubernetes with security, scalability, and ease of use in mind.

Table of Contents

• Basic Usage
• Installation
• Development
  • Makefile
  • Environment reproducibility
    • Editor config
    • Setting environment variables (direnv)
• Versioning
• License

Basic Usage

Note: This is a high-level overview. For more details, please refer to the Basic Usage Docs.

At its core, Poiesis lets you run a single task via a GA4GH TES-compliant API. But since it runs natively on Kubernetes, it’s far more versatile:

• Use Poiesis as the execution backend for workflow engines
• Launch thousands of batch tasks efficiently on Kubernetes
• Train and track ML models using reproducible task definitions

Whether you're running a one-off container or building full workflows, Poiesis abstracts away the complexity—so you can focus on what to run, not how to run it.

Installation

Note: This is a high-level overview. For detailed instructions, please refer to the Deployment Docs.

To install Poiesis into your Kubernetes cluster using Helm:

1. Navigate to the Helm chart directory:

   cd deployment/helm

2. Run the following command to install Poiesis:

   helm install poiesis . -n poiesis --create-namespace

This will install Poiesis into a new namespace called poiesis.

Development

To get started with development:

Note: This is a high-level overview. For detailed instructions, please refer to the Development Docs.

1. Create a virtual environment using uv:

   make v

   Note: You’ll need to install uv first — see their docs for setup instructions.

2. Activate the virtual environment:

   source .venv/bin/activate

   (Optional: alias this command to something like sv for convenience.)

3. Install dependencies:

   make i

4. Start all the services:

   kubectl apply -f ./deployment/dev.yaml

5. Set environment variables:

   • Copy the .envrc.template to .envrc and fill in the necessary values.
   • If you’re using direnv, it will automatically load .envrc.
   • Alternatively, you can load the file manually:

   source .envrc

6. Run the server:

   make dev

Makefile

For ease of use, certain scripts have been abbreviated in Makefile, make sure that you have installed the dependencies before running the commands.

Note: make commands are only available for Unix-based systems.

To view the commands available, run:

make

Here are certain commands that you might find useful:

• Make a virtual environment

make v

• Install all dependencies including optional dependencies

make i

Note: This project uses optional dependency groups such as types, code_quality, docs, vulnerability, test, and misc. To install stubs or types for the dependencies, you must use the following command:

uv add types-foo --group types

Replace types-foo with the name of the package for the types. All runtime dependencies should be added to the default group. For example, to install requests and its type stubs, run:

uv add requests
uv add types-requests --group types

This ensures that the type checker functions correctly.

Note: Since the dependencies are segregated into groups, if you add a new group make sure to add it in make install command in Makefile.

• Run tests

make t

• Run linter, formatter and spell checker

make fl

• Build the documentation

make d

Note: If you make changes to the code, make sure to generate and push the documentation using above command, else the documentation check CI will fail. Do NOT edit auto-generated documentation manually.

• Run type checker

make tc

• Run all pre-commit checks

make pc

Note: This is not the complete list of commands, run make to find out if more have been added.

Environment reproducibility

Editor Config

To ensure a consistent code style across the project, we include an .editorconfig file that defines the coding styles for different editors and IDEs. Most modern editors support this file format out of the box, but you might need to install a plugin for some editors. Please refer to the EditorConfig website.

Setting environment variables (direnv)

Our project uses .envrc files to manage environment variables. Wherever such a file is required across the project tree, you will find a .envrc.template file that contains the necessary variables and helps you create your own personal copy of each file. You can find the locations of all .envrc.template files by executing find . -type f -name \.envrc\.template in the root directory. For each, create a copy named .envrc in the same directory, open it in a text editor and replace the template/example values with your own personal and potentially confidential values.

Warning: Be careful not to leak sensitive information! In particular, never add your secrets to the .envrc.template files directly, as these are committed to version control and will be visible to anyone with access to the repository. Always create an .envrc copy first (capitalization and punctuation matter!), as these (along with .env files) are ignored from version control.

Once you have filled in all of your personal information, you can have the direnv tool manage setting your environment variables automatically (depending on the directory you are currently in and the particular .envrc file defined for that directory) by executing the following command:

direnv allow

Versioning

The project adopts the semantic versioning scheme for versioning. Currently the software is in a pre-release stage, so changes to the API, including breaking changes, may occur at any time without further notice.

License

This project is distributed under the Apache License 2.0, a copy of which is also available in LICENSE.