Package Specification and Ecosystem

[claymcleod]

Note

The state of this RFC is a work in progress.

Overview

This RFC proposes the introduction of a package specification and ecosystem that is formally supported within any compliant WDL implementation. It accomplishes this by defining the structure of WDL packages and outlining the mechanisms by which those packages may be imported within consumer WDL workflows.

The following is an example of what importing a task for fastp might look like in a WDL worflow of the future. Here, we'll use the existing BioWDL implementation of fastp.

import fastp from biowdl

workflow hello {
  # ...

  call fastp.Fastp {
    # ...
  }

  # ...
}

[dependencies]
biowdl = { git = "https://github.com/biowdl/tasks", branch = "develop" }

Motivation

Importing in WDL today is a relatively limited operation—far too limited to meet the needs of modern, distributed software development and a package ecosystem. Only local file references or references to concrete URLs are supported during import, neither of which are sufficient for properly supporting a distributed package ecosystem.

• Importing of local files requires the files to be resident, which means you have to actually go out and manually download the packages you want locally to your system before you run your workflow—a logistical nightmare.
• Importing of URLs is fine for simple imports that rarely change. However, it's still a far cry from enabling a thriving ecosystem of versioned packages.
  • Versioning is not inherently baked into or required to be present in URLs, so tasks and workflows are rarely versioned at all. In practice, imports typically just point to the most up-to-date document on a main/master branch. Since no conventions or backwards compatibility guarantees are made, documents can easily change out from under you at any time.
  • In cases where the package versions are baked into the URL (for example, pointing to WDL documents on a particular branch or tag in at https://raw.githubusercontent.com), one can implement some level of versioned importing. The result is still undesirable though, as the version of dependencies is baked into the source code itself. This means that a change from v2.0.1 to v2.0.2 of a dependency means that all URLs pointing to this dependency need to be updated. This is unnecessary work and added noise for the git commit history.

All of this is made a bit easier through the use of relative imports (e.g., import "../task.wdl"), which ensure that imports are resolved relative the current document location. This means that, whether you run a workflow from a local document (sprocket run ./workflow.wdl) or a remote document (sprocket run https://example.com/workflow.wdl`), imports will sourced correctly. That being said, this does not solve the issue of importing from external packages.

A more ideal state

We consider the following requirements of a solid foundation upon which to build a package ecosystem.

• Symbolic resolution of package names to their defined source should be supported across execution engines. The file format for defining the dependencies is defined by the WDL specification, and the lookup/resolution behavior should similarly be dictated by the specification.
• Imports should be supported across a broader set of modalities that better reflect the current state of software engineering best practices including,
  • Symbolically importing local directories.
  • Symbolically importing URLs.
  • Symbolically importing from git (new), with support for pulling a specific tag, branch, revision, and path.
• Creation and maintenance of packages should be straightforward such that it is the norm for developing WDL codebases. This means that each package is versioned, and those versions should be required to conform to some community standard, such as semantic versioning so that tooling around package resolution and management can rely on the guarantees therein.

Goals

Given the above ideal state, this RFC sets out the following goals.

• Provide a straightforward mechanism for importing and including external tasks and sub-workflows in a subsequent downstream workflow.
• Enable distributed responsibility for the development and maintenance of packages (i.e., it should be designed around single packages repository, though those are welcome to be created as well). The end goal is that tool authors themselves should be incentivized to include alongside their tool a high-quality WDL package that can be easily imported, and the tools that do that well have a higher chance of being used in WDL.
• Allow flexibility in the various access patterns for that code importing (e.g., by a released/tagged version, by a branch, by a commit).
• Guarantees regarding backwards compatibility and some level of immutability should be present.
• Facilitate automated tools that enforce proper software development lifecycle practices to be developed, which an eye towards (a) pinning of packages to a particular version, (b) straightforward package resolution, (c) automated package upgrading, and (d) communication of the license(s) for dependencies in a machine-readable manner (e.g., via a SPDX license identifier).
• Renaming of imported items should be supported to avoid naming conflicts.

Antigoals

• This RFC is not intended to a single package repository or even a set of centralized, best practices workflows (a la nf-core)—in fact, quite the contrary! Instead, it leans into and enforces one of WDL's core differentiators, which is a focus on enabling distributed development by a number of disparate teams and individuals. In other words, we aim to create an environment where tool authors themselves can compete to create and make available their own WDL packages alongside their tools, and workflow authors can easily identify these packages for inclusion in their workflows. We believe that focusing on building a distributed system where authors are incentivized to maintain their tools themselves is likely to be far more scalable than creating a centralized, curated platform.

Prior Art

#226

As far as I can tell, this is the earliest official proposal to add some level of package management/versioning to WDL. The issue is rather short and describes a syntax like the following.

import "tools/published_tool" @ "v1.0"
import "tools/new_tool_version" @ "v11.3"
import "structs/alpha_struct_defs" @ "v0.5"

As previously stated, I'm not in support of stopping at the proposed mechanism because it assumes a centralized package system (something I don't think is feasible with our relatively small contributor team nor advisable based on our emphasis in enabling distributed development and maintenance).

#493

The revival of this idea in early 2022, this proposal similarly focuses on the details of a centralized package repository. For the pieces that overlap with this proposal (essentially just the syntax of how this importing would work), this proposal and #493 overlap quite a bit—just a difference in the order of the "import" and "from" clauses.

#499

A few months later, some work was done to outline what the package format might look like concretely. Some inspiration was taken from this work regarding the information that should be collected (name, author, version, license). Further, the concerns regarding reproducibility were included in the goals of this proposal where needed. For the remainder of text in that proposal, the fact that we are relying so heavily on distributed package management with git rather than a centralized package store makes much of the conversation irrelevant to this proposal (e.g., tar vs zip).

#698

#698 proposes that we relax the constraints around what document versions can be imported, essentially advocating that any WDL v1.x version must be able to load any WDL documents with versions v1.x or lower. This adheres to common expectations regarding backwards compatibility of software with the same major version and would be required for this proposal to be manageable (else, the entire ecosystem could become deadlocked waiting for root packages in the ecosystem to update their WDL version when a new minor revision of WDL is released).

Proposal

Note

This section specifically uses the sprocket engine that my team is writing as an example. This is because it's hard to make a concrete case for what a solution might look like outside of both (a) the WDL specification and (b) a concrete execution engine.

Overview

This RFC proposed the following actions: all of which get their own sections below.

• Define what form a WDL package takes.
• Introduce syntax for symbolic imports.
• Introduce requirements on execution engines for these imports.
• (Optionally) Deprecate the importing of individual files.

Definition of a WDL package

The ability to create a dependency tree of packages requires defining a manifest file that describes the content of each package and all of it's dependencies. To that end, a WDL package is defined as a directory containing a manifest file and zero or more WDL files.

Manifest file

The manifest file for a package always lives at the root of the directory with the name wdl-package.json. The manifest file contains the following keys.

• authors. The authors field is an array of strings where each string is an author description with no enforced format. The convention of singular authors should be "First Last <first.last@example.com>", but this is not a requirement.
• version. The version field is a string that adheres to the SemVer v2.0.0 specification with respect to backwards compatibility guarantees and versioning expectations.
• license. The license field is a string that defines the license requirements of the package. It contains a list of SPDX license identifiers concatenated with AND and OR. In more complex cases, parentheses can be used to group items and establish associativity.
• dependencies. The dependencies field is a JSON object that contains one key per import for that package. Each key must be a valid WDL identifier. The value of the field can be one of the following.
  • A string that is interpreted as a file path, a URL, or a GitHub repository.
  • A JSON object that defines a more complex import object.

Here would be an example of a wdl-package.json file.

{
  "authors": ["Clay McLeod <clay.mcleod@stjude.org>"],
  "version": "0.1.0",
  "license": "MIT OR Apache-2.0",
  "dependencies": {
    "samtools": "~/samtools", // A local file.
    "bwa": "https://example.com/bwa", // A URL.
    "biowdl": "biowdl/tasks", // A Github repository.
    "special": {
      "git": "https://another.git/repository.git",
      "path": "a/path/underneath/the/git/repository",
      "branch": "develop"
    }
  }
}

These can then be imported using the following syntax.

import Foo from samtools
import Bar from bwa
import Baz from biowdl
import Quux from special

Name field

Notably, the name field is missing from the manifest file. This is an intentional decision, as packages and imported directly from source and not from a package management server where they have an identifier. This fact removes the need to capture a name, as the importer simply names the package whatever they want to refer to it by (as long as it's a valid WDL identifier) in their wdl-package.json.

Other notes

• JSON was chosen as the file format because the format is already associated with the WDL specification and is readily used to express configuration information.
• The name wdl-package.json was chosen to ensure there is no conflict with the

Warning

This RFC is being rewritten and I've stopped here. Content below this point is currently out of date.

Introduce syntax for symbolic imports

First, let's address the relatively straightforward question of what for the imports would take. Because imports today must be enclosed in quotes, a backwards compatible change that seems sufficient would be that any non-quoted import is assumed to be a symbolic import. For example,

# This is assumed to be a local file import.
import "biowdl"

# This is assumed to be a symbolic import.
import biowdl

Next, there are a number of different ways you can structure the identifier that symbolic imports link into—particularly as it relates to conflicts in the identifiers. To start, we suggest to start with the simplest approach, which is an identifier conforming to a word pattern ([a-zA-Z0-9_]+). The lack of namespacing means that everything must be explicitly handled by the user.

Consider a Sprocket.toml file that contains a dependency section like so:

# ...

[dependencies]
# Importing a package with the same name as the scope.
tasks = { git = "https://github.com/stjudecloud/tasks", tag = "v1.0.0" }

# An import that is explicitly renamed by the user. Notably, under other paradigms, 
# this would cause a name conflict with the import above.
biowdl = { git = "https://github.com/biowdl/tasks", branch = "develop" }

Effectively, we're empowering (and requiring) users to solve name conflicts themselves. The authors feel this a reasonable first step in this direction rather than, say, a complicated set of name conflict requirements.

Concretely, the following import modalities would be required (as show through examples in the fictional Sprocket.toml file).

[dependencies]
# Importing packages on the local filesystem is required. 
# No further options are specified—the files are just imported as is.
package = { path = "../../path/to/my/package" }

# Importing packages from a public git repository is required.
# Packages in this form _must_ specify the branch, tag, or commit that 
# needs to be checked out.
package = { git = "https://github.com/package/name", branch = "main" }
package = { git = "https://github.com/package/name", tag = "v1.0.0" }
package = { git = "https://github.com/package/name", commit = "bcd123c" }

Notably, the specifics of how these imports would be linked in across the various execution engines is left up to the execution engine—there is no requirement from the specification other than (a) these modalities must be supported and (b) they must be clearly documented for users to be able to take advantage of.

Concerns left to address.

• Should exported items have an associated visibility?
• Cascaded importing of WDL through the dependency tree while also decoupling it from the source code.