diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md deleted file mode 100644 index 8dfbcb08fc..0000000000 --- a/docs/developer/plugins.md +++ /dev/null @@ -1,545 +0,0 @@ -(plugins-dev-page)= - -# Plugins - -This page describes how to create, test, and publish third-party plugins. - -## Plugin structure - -The best way to get started with your own plugin is to refer to the [nf-hello](https://github.com/nextflow-io/nf-hello) repository. This repository provides a minimal plugin implementation with several examples of different extension points and instructions for building, testing, and publishing. - -Plugins can be written in Java or Groovy. - -The minimal dependencies are as follows: - -```groovy -dependencies { - compileOnly project(':nextflow') - compileOnly 'org.slf4j:slf4j-api:1.7.10' - compileOnly 'org.pf4j:pf4j:3.4.1' - - testImplementation project(':nextflow') - testImplementation "org.codehaus.groovy:groovy:4.0.24" - testImplementation "org.codehaus.groovy:groovy-nio:4.0.23" -} -``` - -The plugin subproject directory name must begin with the prefix `nf-` and must include a file named `src/resources/META-INF/MANIFEST.MF` which contains the plugin metadata. The manifest content looks like the following: - -``` -Manifest-Version: 1.0 -Plugin-Class: the.plugin.ClassName -Plugin-Id: the-plugin-id -Plugin-Provider: Your Name or Organization -Plugin-Version: 0.0.0 -``` - -## Extension points - -Nextflow's plugin system exposes a variety of extension points for plugins. This section describes how to use these extension points when writing a plugin, as well as how they are used in a pipeline. - -:::{note} -If you would like to implement something in a plugin that isn't covered by any of the following sections, feel free to create an issue on GitHub and describe your use case. In general, any class in the Nextflow codebase that implements `ExtensionPoint` can be extended by a plugin, and existing plugins are a great source of examples when writing new plugins. -::: - -:::{note} -Plugin extension points must be added to `extensions.idx` in the plugin repository to make them discoverable. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) plugin for an example. -::: - -### Commands - -Plugins can define custom CLI commands that can be executed with the `nextflow plugin` command. - -To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entrypoint (the class that extends `BasePlugin`). Alternatively, you can implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement two methods, `getCommands()` and `exec()`: - -```groovy -import nextflow.cli.PluginAbstractExec -import nextflow.plugin.BasePlugin - -class MyPlugin extends BasePlugin implements PluginAbstractExec { - @Override - List getCommands() { - [ 'hello' ] - } - - @Override - int exec(String cmd, List args) { - if( cmd == 'hello' ) { - println "Hello! You gave me these arguments: ${args.join(' ')}" - return 0 - } - else { - System.err.println "Invalid command: ${cmd}" - return 1 - } - } -} -``` - -You can then execute this command using the `nextflow plugin` command: - -```bash -nextflow plugin my-plugin:hello --foo --bar -``` - -See the {ref}`cli-plugin` CLI command for usage information. - -### Configuration - -Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option in a safe manner -- if the option isn't defined, it will return `null`. A common practice is to define any configuration for your plugin in a custom config scope. - -For example, you can query a config option in a trace observer hook: - -```groovy -import nextflow.Session -import nextflow.trace.TraceObserver - -class MyObserver implements TraceObserver { - - @Override - void onFlowCreate(Session session) { - final message = session.config.navigate('myplugin.createMessage') - println message - } -} -``` - -You can then set this option in your config file: - -```groovy -// dot syntax -myplugin.createMessage = "I'm alive!" - -// block syntax -myplugin { - createMessage = "I'm alive!" -} -``` - -:::{versionadded} 25.02.0-edge -::: - -Plugins can declare their config options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation: - -```groovy -import nextflow.config.schema.ConfigOption -import nextflow.config.schema.ConfigScope -import nextflow.config.schema.ScopeName -import nextflow.script.dsl.Description - -@ScopeName('myplugin') -@Description(''' - The `myplugin` scope allows you to configure the `nf-myplugin` plugin. -''') -class MyPluginConfig implements ConfigScope { - - MyPluginConfig(Map opts) { - this.createMessage = opts.createMessage - } - - @ConfigOption - @Description('Message to print to standard output when a run is initialized.') - String createMessage -} -``` - -While this approach is not required to support plugin config options, it allows Nextflow to recognize plugin definitions when validating the config. - -### Executors - -Plugins can define custom executors that can then be used with the `executor` process directive. - -To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor: - -```groovy -import nextflow.executor.Executor -import nextflow.util.ServiceName -import org.pf4j.ExtensionPoint - -@ServiceName('my-executor') -class MyExecutor extends Executor implements ExtensionPoint { - - // ... - -} -``` - -You can then use this executor in your pipeline: - -```nextflow -process foo { - executor 'my-executor' - - // ... -} -``` - -:::{tip} -Refer to the source code of Nextflow's built-in executors to see how to implement the various components of an executor. You might be able to implement most of your executor by simply reusing existing code. -::: - -### Filesystems - -Plugins can define custom filesystems that can be used by Nextflow to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. - -To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem: - -```groovy -import java.nio.file.spi.FileSystemProvider - -class MyFileSystemProvider extends FileSystemProvider { - - @Override - String getScheme() { - return 'myfs' - } - - // ... -} -``` - -You can then use this filesystem in your pipeline: - -```nextflow -input = file('myfs://path/to/input/file.txt') -``` - -See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information. Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. - -:::{tip} -Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3. -::: - -### Functions - -:::{versionadded} 22.09.0-edge -::: - -Plugins can define custom functions, which can then be included in Nextflow pipelines. - -To implement a custom function, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Function` annotation: - -```groovy -import nextflow.Session -import nextflow.plugin.extension.Function -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Function - String reverseString(String origin) { - origin.reverse() - } - -} -``` - -You can then use this function in your pipeline: - -```nextflow -include { reverseString } from 'plugin/my-plugin' - -channel.of( reverseString('hi') ) -``` - -You can also use an alias: - -```nextflow -include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' -``` - -### Operators - -:::{versionadded} 22.04.0 -::: - -Plugins can define custom channel factories and operators, which can then be included into Nextflow pipelines. - -To implement a custom factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Factory` or `Operator` annotation: - -```groovy -import groovyx.gpars.dataflow.DataflowReadChannel -import groovyx.gpars.dataflow.DataflowWriteChannel -import nextflow.Session -import nextflow.plugin.extension.Factory -import nextflow.plugin.extension.Operator -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Factory - DataflowWriteChannel fromQuery(Map opts, String query) { - // ... - } - - @Operator - DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { - // ... - } - -} -``` - -You can then use them in your pipeline: - -```nextflow -include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' - -def sql = 'select * from FOO' -channel - .fromTable(sql, db: 'test', emitColumns: true) - .sqlInsert(into: 'BAR', columns: 'id', db: 'test') -``` - -The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory -is included under the alias `fromTable`. - -### Process directives - -Plugins that implement a [custom executor](#executors) will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task's configuration. As a best practice, custom executors should try to support all process directives that have executor-specific behavior and are relevant to your executor. - -Nextflow does not provide the ability to define custom process directives in a plugin. Instead, you can use the {ref}`process-ext` directive to provide custom process settings to your executor. Try to use specific names that are not likely to conflict with other plugins or existing pipelines. - -For example, a custom executor can use existing process directives as well as a custom setting through the `ext` directive: - -```groovy -class MyExecutor extends Executor { - - @Override - TaskHandler createTaskHandler(TaskRun task) { - final cpus = task.config.cpus - final memory = task.config.memory - final myOption = task.config.ext.myOption - - println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" - - // ... - } - - // ... - -} -``` - -### Trace observers - -:::{versionchanged} 25.04 -The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead. -::: - -A *trace observer* in Nextflow is an entity that can listen and react to workflow events, such as when a workflow starts, a task completes, a file is published, etc. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. - -Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver`, and implement the `create()` method in your observer factory: - -```groovy -// MyObserverFactory.groovy -import nextflow.Session -import nextflow.trace.TraceObserver -import nextflow.trace.TraceObserverFactory - -class MyObserverFactory implements TraceObserverFactory { - - @Override - Collection create(Session session) { - final enabled = session.config.navigate('myplugin.enabled') - return enabled ? [ new MyObserver() ] : [] - } -} - -// MyObserver.groovy -import java.nio.file.Path - -import nextflow.processor.TaskHandler -import nextflow.trace.TraceObserver -import nextflow.trace.TraceRecord - -class MyObserver implements TraceObserver { - - @Override - void onFlowBegin() { - println "Okay, let's begin!" - } - - @Override - void onProcessComplete(TaskHandler handler, TraceRecord trace) { - println "I completed a task! It's name is '${handler.task.name}'" - } - - @Override - void onProcessCached(TaskHandler handler, TraceRecord trace) { - println "I found a task in the cache! It's name is '${handler.task.name}'" - } - - @Override - void onFilePublish(Path destination, Path source) { - println "I published a file! It's located at ${path.toUriString()}" - } - - @Override - void onFlowError(TaskHandler handler, TraceRecord trace) { - println "Uh oh, something went wrong..." - } - - @Override - void onFlowComplete() { - println 'All done!' - } -} -``` - -You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: - -```groovy -myplugin.enabled = true -``` - -Refer to the `TraceObserver` [source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. - -## Publishing plugins - -Nextflow resolves plugins through a plugin registry, which stores metadata for each plugin version, including the publishing date, checksum, and download URL for the plugin binary. The default registry is located on GitHub at [nextflow-io/plugins](https://github.com/nextflow-io/plugins/), specifically [this file](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -To publish a plugin, you must create the plugin release, publish it to GitHub, and make a pull request against the main plugin registry with the metadata for your plugin release. - -A plugin release is a ZIP archive containing the compiled plugin classes, the required dependencies, and a JSON file with the plugin metadata. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) example to see how to create a plugin release. - -Here is an example meta file for a plugin release: - -```json -{ - "version": "0.2.0", - "url": "https://github.com/nextflow-io/nf-amazon/releases/download/0.2.0/nf-amazon-0.2.0.zip", - "date": "2020-10-12T10:05:44.28+02:00", - "sha512sum": "9e9e33695c1a7c051271..." -} -``` - -:::{note} -The plugin version should be a valid [semantic version](https://semver.org/). -::: - -(testing-plugins)= - -## Testing plugins - -Plugins must be declared in the `nextflow.config` file using the `plugins` scope, for example: - -```groovy -plugins { - id 'nf-amazon@0.2.0' -} -``` - -If a plugin is not locally available, Nextflow checks the repository index for the download URL, downloads and extracts the plugin archive, and installs the plugin into the directory specified by `NXF_PLUGINS_DIR` (default: `${NXF_HOME}/plugins`). - -Since each Nextflow run can have a different set of plugins (and versions), each run keeps a local plugins directory called `.nextflow/plr/` which links the exact set of plugins required for the given run. - -Additionally, the "default plugins" (defined in the Nextflow resources file `modules/nextflow/src/main/resources/META-INF/plugins-info.txt`) are always made available for use. To disable the use of default plugins, set the environment variable `NXF_PLUGINS_DEFAULT=false`. - -When running in development mode, the plugin system uses the `DevPluginClasspath` to load plugin classes from each plugin project build path, e.g. `plugins/nf-amazon/build/classes` and `plugins/nf-amazon/build/target/libs` (for deps libraries). - -## Environment variables - -The following environment variables are available when developing and testing plugins: - -`NXF_PLUGINS_MODE` -: The plugin execution mode, either `prod` for production or `dev` for development (see below for details). - -`NXF_PLUGINS_DIR` -: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production, `./plugins` in development). - -`NXF_PLUGINS_DEFAULT` -: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: `true`). - -`NXF_PLUGINS_DEV` -: Comma-separated list of development plugin root directories. - -`NXF_PLUGINS_TEST_REPOSITORY` -: :::{versionadded} 23.04.0 - ::: -: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. -: The URI should refer to a plugins repository JSON file or a specific plugin JSON meta file. In the latter case it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. - -: For example: - - ```bash - # custom plugin repository at https://github.com/my-org/plugins - export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" - - # custom plugin release - export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" - - nextflow run -plugins nf-hello - ``` - -: This variable is useful for testing a plugin release before publishing it to the main registry. - -## Gradle Tasks - -The following build tasks are defined in the `build.gradle` of each core plugin as well as the `nf-hello` example plugin. - -### `makeZip` - -Create the plugin archive and JSON meta file in the subproject `build/libs` directory. - -```console -$ ls -l1 plugins/nf-tower/build/libs/ -nf-tower-0.1.0.jar -nf-tower-0.1.0.json -nf-tower-0.1.0.zip -``` - -### `copyPluginLibs` - -Copy plugin dependencies JAR files into the subproject `build/target/libs` directory. This is needed only when launching the plugin in development mode. - -### `copyPluginZip` - -Copy the plugin archive to the root project build directory, i.e. `build/plugins`. - -### `uploadPlugin` - -Upload the plugin archive and meta file to the corresponding GitHub repository. Options: - -`release` -: The plugin version, e.g. `1.0.1`. - -`repo` -: The GitHub repository name, e.g. `nf-amazon`. - -`owner` -: The GitHub owning organization, e.g. `nextflow-io`. - -`skipExisting` -: Do not upload a file if it already exists, i.e. checksum is the same (default: `true`). - -`dryRun` -: Execute the tasks without uploading file (default: `false`). - -`overwrite` -: Prevent to overwrite a remote file already existing (default: `false`). - -`userName` -: The user name used to authenticate GitHub API requests. - -`authToken` -: The personal token used to authenticate GitHub API requests. - -### `upload` - -Upload the plugin archive and meta file. - -### `publishIndex` - -Upload the plugins index to the repository hosted at [nextflow-io/plugins](https://github.com/nextflow-io/plugins), which makes them publicly accessible at [this URL](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -## Additional Resources - -* [PF4J](https://pf4j.org/) -* [Understanding Gradle: The Build Lifecycle](https://proandroiddev.com/understanding-gradle-the-build-lifecycle-5118c1da613f) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md new file mode 100644 index 0000000000..2f9277e898 --- /dev/null +++ b/docs/gradle-plugin.md @@ -0,0 +1,82 @@ +(gradle-plugin-page)= + +# Using the Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. + +:::{note} +The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. +::: + +(gradle-plugin-create)= + +## Creating a plugin + +:::{versionadded} 25.04.0 +::: + +The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. + +To create a Nextflow plugin with the Gradle plugin, run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. + +See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. See {ref}`dev-plugins-extension-points` for more information about using plugin extension points. + +## Building a plugin + +To build a plugin, run `make assemble`. + +Plugins can also be installed locally without being published. To install a plugin locally: + +1. In the plugin root directory, run `make install`. + + :::{note} + Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. + ::: + +2. Run your pipeline: + + ```bash + nextflow run main.nf -plugins @ + ``` + + :::{note} + Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. + ::: + +(gradle-plugin-test)= + +## Testing a plugin + +

Unit tests

+ +Unit tests are small, focused tests designed to verify the behavior of individual plugin components. + +To run unit tests: + +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example unit test. + +2. In the plugin root directory, run `make test`. + +

End-to-end tests

+ +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. + +(gradle-plugin-publish)= + +## Publishing a plugin + +The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. + +To publish your plugin: + +1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. + +2. Add the following properties: + + ``` + pluginRegistry.accessToken= + ``` + + Replace `` with your plugin registry access token. + +3. Run `make release`. diff --git a/docs/index.md b/docs/index.md index 2014a48e5d..fe9a8101b6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -67,7 +67,6 @@ config executor cache-and-resume reports -plugins ``` ```{toctree} @@ -145,6 +144,16 @@ strict-syntax migrations/index ``` +```{toctree} +:hidden: +:caption: Plugins +:maxdepth: 1 + +plugins/plugins +plugins/using-plugins +plugins/developing-plugins +``` + ```{toctree} :hidden: :caption: Contributing @@ -153,7 +162,6 @@ migrations/index developer/index developer/diagram developer/packages -developer/plugins ``` ```{toctree} @@ -162,6 +170,8 @@ developer/plugins :maxdepth: 1 data-lineage +gradle-plugin +migrating-plugin-registry updating-spot-retries metrics flux diff --git a/docs/migrating-plugin-registry.md b/docs/migrating-plugin-registry.md new file mode 100644 index 0000000000..3d5291365f --- /dev/null +++ b/docs/migrating-plugin-registry.md @@ -0,0 +1,160 @@ +(migrating-plugin-page)= + +# Migrating to the Nextflow plugin registry + +The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. + +:::{note} +The Nextflow plugin registry and Gradle plugin are currently available as a private beta. Plugin developers are encouraged to contact [info@nextflow.io](mailto:info@nextflow.io) for more information about accessing the registry. +::: + +## Overview + +### Nextflow plugin registry + +The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. Nextflow 25.04 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. + +### Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. + +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. + +## Timeline + +The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprecated in favor of the Nextflow plugin registry. + +:::{note} +The following timeline is tentative and subject to modification. +::: + +

Nextflow 25.04

+ +The Nextflow plugin registry is available as a private beta. Nextflow 25.04 can use the Nextflow plugin registry as an opt-in feature. The Nextflow plugin registry will be automatically kept up-to-date with the [legacy plugin index](https://github.com/nextflow-io/plugins). + +During this time, plugin developers are encouraged to experiment with the Gradle plugin and plugin registry. + +

Nextflow 25.10

+ +The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. + +Developers will be required to publish to the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. + +

Nextflow 26.04

+ +Nextflow 26.04 will only be able to use the Nextflow plugin registry. + +At some point in the future, the legacy plugin index will be **frozen** -- it will no longer receives updates from the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will remain available indefinitely. + +## Impact on plugin users + +No immediate actions are required for plugin users. The plugin configuration has not changed. + +## Impact on plugin developers + +Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The easiest way to do this is to migrate to the Nextflow Gradle plugin, which simplifies the development process and supports publishing to the plugin registry from the command line. + +### Migrating to the Nextflow Gradle plugin + +To migrate an existing Nextflow plugin: + +1. Remove the following files and folders: + - `buildSrc/` + - `launch.sh` + - `plugins/build.gradle` + +2. If your plugin has a `plugins` directory, move the `src` directory to the project root. + + :::{note} + Plugin sources should be in `src/main/groovy` or `src/main/java`. + ::: + +3. Replace the contents of `settings.gradle` with the following: + + ```groovy + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. In the project root, create a new `build.gradle` file with the following configuration: + + ```groovy + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + registry { + authToken = project.findProperty('pluginRegistry.accessToken') + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.04.0`. + - `PROVIDER`: Your name or organization—for example, `acme`. + - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. + +5. Replace the contents of `Makefile` with the following: + + ```Makefile + # Build the plugin + assemble: + ./gradlew assemble + + clean: + rm -rf .nextflow* + rm -rf work + rm -rf build + ./gradlew clean + + # Run plugin unit tests + test: + ./gradlew test + + # Install the plugin into local nextflow plugins dir + install: + ./gradlew install + + # Publish the plugin + release: + ./gradlew releasePlugin + ``` + +6. Update `README.md` with information about the structure of your plugin. + +7. In the plugin root directory, run `make assemble`. + +Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template and add your existing plugin code. See {ref}`dev-plugins-template` for more information about the plugin template. + +### Publishing to the Nextflow plugin registry + +The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information. + +Once you migrate to the Gradle plugin, you will no longer be able to publish to the legacy plugin index. See the [transition timeline](#timeline) for more information. diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 20ea621c3f..2c10bd1579 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -46,6 +46,16 @@ You can explore this lineage from the command line using the {ref}`cli-lineage` See the {ref}`data-lineage-page` guide to get started. +

Simplified plugin development

+ +The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. + +See the {ref}`gradle-plugin-page` guide for details. + +:::{note} +The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. +::: + ## Enhancements

Improved inspect command

diff --git a/docs/plugins.md b/docs/plugins.md deleted file mode 100644 index 68d33151c1..0000000000 --- a/docs/plugins.md +++ /dev/null @@ -1,63 +0,0 @@ -(plugins-page)= - -# Plugins - -Nextflow has a plugin system that allows the use of extensible components that are downloaded and installed at runtime. - -(plugins-core)= - -## Core plugins - -The following functionalities are provided via plugin components, and they make part of the Nextflow *core* plugins: - -- `nf-amazon`: Support for Amazon Web Services. -- `nf-azure`: Support for Microsoft Azure. -- `nf-cloudcache`: Support for the cloud cache (see `NXF_CLOUDCACHE_PATH` under {ref}`config-env-vars`). -- `nf-console`: Implement Nextflow [REPL console](https://www.nextflow.io/blog/2015/introducing-nextflow-console.html). -- `nf-k8s`: Support for Kubernetes. -- `nf-google`: Support for Google Cloud. -- `nf-tower`: Support for [Seqera Platform](https://seqera.io) (formerly Tower Cloud). -- `nf-wave`: Support for [Wave containers](https://seqera.io/wave/) service. - -## Using plugins - -The core plugins do not require any configuration. They are automatically installed when the corresponding feature is requested by a Nextflow pipeline. You can still specify them as described below, e.g. if you want to pin the version of a plugin, however if you try to use a plugin version that isn't compatible with your Nextflow version, Nextflow will fail. - -You can enable a plugin by declaring it in your Nextflow configuration: - -```groovy -plugins { - id 'nf-hello@0.1.0' -} -``` - -Or you can use the `-plugins` command line option: - -```bash -nextflow run -plugins nf-hello@0.1.0 -``` - -The plugin identifier consists of the plugin name and plugin version separated by a `@`. Multiple plugins can be specified in the configuration with multiple `id` declarations, or on the command line as a comma-separated list. When specifying plugins via the command line, any plugin declarations in the configuration file are ignored. - -The plugin version is optional. If it is not specified, Nextflow will download the latest plugin version that is compatible with your Nextflow version. In general, it recommended that you not specify the plugin version unless you actually want to stick to that version, such as for [offline usage](#offline-usage). - -The core plugins are documented in this documentation. For all other plugins, please refer to the plugin's code repository for documentation and support. - -:::{versionadded} 25.02.0-edge -::: - -The plugin version can be prefixed with `~` to pin the major and minor version while allowing the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`, which is `2.9.2`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. - -## Offline usage - -To use Nextflow plugins in an offline environment: - -1. {ref}`Install Nextflow ` on a system with an internet connection. - -2. Download any additional plugins by running `nextflow plugin install `. Alternatively, simply run your pipeline once and Nextflow will download all of the plugins that it needs. - -3. Copy the `nextflow` binary and `$HOME/.nextflow` folder to your offline environment. - -4. In your Nextflow configuration file, specify each plugin that you downloaded, both name and version, including default plugins. This will prevent Nextflow from trying to download newer versions of plugins. - -Nextflow caches the plugins that it downloads, so as long as you keep using the same Nextflow version and pin your plugin versions in your config file, Nextflow will use the locally installed plugins and won't try to download them from the Internet. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md new file mode 100644 index 0000000000..c48417660b --- /dev/null +++ b/docs/plugins/developing-plugins.md @@ -0,0 +1,516 @@ +(dev-plugins-page)= + +# Developing plugins + +This page describes how to develop plugins for Nextflow. + +(dev-plugins-template)= + +## Nextflow plugin template + +The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It uses [Gradle](https://gradle.org/), a build automation tool optimized for Java and Groovy projects, as well as the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle). + +You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. + +:::{note} +The Nextflow Gradle plugin is currently available as a private beta. See the {ref}`migration guide ` for more information. +::: + +### Structure + +The plugin template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. + +For example, a plugin created from the plugin template with the name `nf-hello` and organization `nextflow` will have the following structure: + +```console +nf-hello +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ ├── gradle-wrapper.jar +│ └── gradle-wrapper.properties +├── gradlew +├── settings.gradle +├── src +│ ├── main +│ │ └── groovy +│ │ └── nextflow +│ │ └── hello +│ │ ├── HelloExtension.groovy +│ │ ├── HelloFactory.groovy +│ │ ├── HelloObserver.groovy +│ │ └── HelloPlugin.groovy +│ └── test +│ └── groovy +│ └── nextflow +│ └── hello +│ └── HelloObserverTest.groovy +└── validation + └── main.nf + └── nextflow.config +``` + +This structure contains the following key files and folders: + +- `.github/workflows`: GitHub Action which implements continuous integration for the plugin. + +- `build.gradle`: The Gradle build script. + +- `COPYING`: The project license, detailing the terms under which the code can be used and distributed.​ + +- `gradle/wrapper/`: Helper files for the Gradle Wrapper. + +- `gradlew`: The Gradle Wrapper script, which allows you to use Gradle without installing it into your environment. + +- `Makefile`: Defines common tasks for building, testing, and publishing the plugin with Make.​ + +- `README.md`: The project README, which provides an overview of the project, including its purpose, features, and instructions for usage and development.​ + +- `settings.gradle`: The Gradle project configuration, which specifies project-specific settings such as the project name and included modules. + +- `src/main/groovy///`: The main source directory, which contains the plugin source code and resources.​ + +- `src/test/groovy///`: The test source directory, which contains the plugin unit tests. + +- `validation`: A small Nextflow pipeline which serves as an end-to-end test for the plugin. + +The plugin template also implements the following example features: + +- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`). + +- A custom function called `sayHello` (see `HelloExtension`). + +### Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. + +It is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), and can be declared and managed like any other dependency in the `build.gradle` file: + +```nextflow +plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' +} +``` + +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +### Make commands + +The plugin template includes a Makefile which wraps the most important Gradle tasks provided by the Nextflow Gradle plugin. + +These tasks can be executed with [Make](https://www.gnu.org/software/make/). For example: + +```bash +make assemble +``` + +The following `make` commands are available: + +`assemble` +: Compiles the Nextflow plugin code and assembles it into a zip archive. + +`install` +: Installs the plugin into the local Nextflow plugins directory. + +`release` +: Publishes the plugin. See {ref}`gradle-plugin-publish` for more information. + +`test` +: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. + +(dev-plugins-extension-points)= + +## Extension points + +Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how to use them. + +### Commands + +Plugins can define custom CLI commands that are executable with the `nextflow plugin` command. + +To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entry point (the class that extends `BasePlugin`). Alternatively, implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement the `getCommands()` and `exec()` methods. For example: + +```groovy +import nextflow.cli.PluginAbstractExec +import nextflow.plugin.BasePlugin + +class MyPlugin extends BasePlugin implements PluginAbstractExec { + @Override + List getCommands() { + [ 'hello' ] + } + + @Override + int exec(String cmd, List args) { + if( cmd == 'hello' ) { + println "Hello! You gave me these arguments: ${args.join(' ')}" + return 0 + } + else { + System.err.println "Invalid command: ${cmd}" + return 1 + } + } +} +``` + +The command can be run using the `nextflow plugin` command: + +```bash +nextflow plugin my-plugin:hello --foo --bar +``` + +See the {ref}`cli-plugin` for usage information. + +### Configuration + +Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option safely. If the option isn’t defined, it will return null. + +A common practice is to use a custom config scope to define any configuration for your plugin. For example: + +```groovy +import nextflow.Session +import nextflow.trace.TraceObserver + +class MyObserver implements TraceObserver { + + @Override + void onFlowCreate(Session session) { + final message = session.config.navigate('myplugin.createMessage') + println message + } +} +``` + +This option can then be set in your configuration file: + +```groovy +// dot syntax +myplugin.createMessage = "I'm alive!" + +// block syntax +myplugin { + createMessage = "I'm alive!" +} +``` + +:::{versionadded} 25.02.0-edge +::: + +Plugins can declare their configuration options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation. For example: + +```groovy +import nextflow.config.schema.ConfigOption +import nextflow.config.schema.ConfigScope +import nextflow.config.schema.ScopeName +import nextflow.script.dsl.Description + +@ScopeName('myplugin') +@Description(''' + The `myplugin` scope allows you to configure the `nf-myplugin` plugin. +''') +class MyPluginConfig implements ConfigScope { + + // no-arg constructor is required to enable validation of config options + MyPluginConfig() { + } + + MyPluginConfig(Map opts) { + this.createMessage = opts.createMessage + } + + @ConfigOption + @Description('Message to print to standard output when a run is initialized.') + String createMessage +} +``` + +This approach is not required to support plugin config options. However, it allows Nextflow to recognize plugin definitions when validating the configuration. + +### Executors + +Plugins can define custom executors that can be used with the `executor` process directive. + +To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: + +```groovy +import nextflow.executor.Executor +import nextflow.util.ServiceName +import org.pf4j.ExtensionPoint + +@ServiceName('my-executor') +class MyExecutor extends Executor implements ExtensionPoint { + + // ... + +} +``` + +You can then use this executor in your pipeline: + +```groovy +process foo { + executor 'my-executor' + + // ... +} +``` + +:::{tip} +See the source code of Nextflow's built-in executors for examples of how to implement various executor components. +::: + +### Filesystems + +Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. + +To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: + +```groovy +import java.nio.file.spi.FileSystemProvider + +class MyFileSystemProvider extends FileSystemProvider { + + @Override + String getScheme() { + return 'myfs' + } + + // ... +} +``` + +You can then use this filesystem in your pipeline: + +```nextflow +input = file('myfs://') +``` + +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for an example of a custom filesystem. + +:::{tip} +Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3. +::: + +### Functions + +:::{versionadded} 22.09.0-edge +::: + +Plugins can define custom functions that can be included in Nextflow pipelines. + +To implement a custom function, create a plugin class that extends the `PluginExtensionPoint` class and implement your function with the `Function` annotation. For example: + +```groovy +import nextflow.Session +import nextflow.plugin.extension.Function +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Function + String reverseString(String origin) { + origin.reverse() + } +} +``` + +You can then add this function to your pipeline: + +```nextflow +include { reverseString } from 'plugin/my-plugin' + +channel.of( reverseString('hi') ) +``` + +Alternatively, you can use an alias: + +```nextflow +include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' +``` + +### Operators + +:::{versionadded} 22.04.0 +::: + +Plugins can define custom channel factories and operators that can then be included in pipelines. + +To implement a custom channel factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class and implement your function with the `Factory` or `Operator` annotation. For example: + +```groovy +import groovyx.gpars.dataflow.DataflowReadChannel +import groovyx.gpars.dataflow.DataflowWriteChannel +import nextflow.Session +import nextflow.plugin.extension.Factory +import nextflow.plugin.extension.Operator +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Factory + DataflowWriteChannel fromQuery(Map opts, String query) { + // ... + } + + @Operator + DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { + // ... + } + +} +``` + +You can then use the custom channel factories or operators in your pipeline: + +```nextflow +include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' + +def sql = 'select * from FOO' +channel + .fromTable(sql, db: 'test', emitColumns: true) + .sqlInsert(into: 'BAR', columns: 'id', db: 'test') +``` + +:::{note} +The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +::: + +:::{tip} +Before creating a custom operator, consider whether the operator can be defined as a [function](#functions) that can be composed with existing operators such as `map` or `subscribe`. Functions are easier to implement and can be used anywhere in your pipeline, not just channel logic. +::: + +### Process directives + +Plugins that implement a custom executor will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. + +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the {ref}`process-ext` directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. + +For example, a custom executor can use existing process directives and a custom setting through the `ext` directive: + +```groovy +class MyExecutor extends Executor { + + @Override + TaskHandler createTaskHandler(TaskRun task) { + final cpus = task.config.cpus + final memory = task.config.memory + final myOption = task.config.ext.myOption + + println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" + + // ... + } + + // ... + +} +``` + +### Trace observers + +:::{versionchanged} 25.04 +The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead. +::: + +A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. + +Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example: + +```groovy +import java.nio.file.Path + +import nextflow.processor.TaskHandler +import nextflow.trace.TraceObserver +import nextflow.trace.TraceRecord + +class MyObserver implements TraceObserver { + + @Override + void onFlowBegin() { + println "Okay, let's begin!" + } + + @Override + void onProcessComplete(TaskHandler handler, TraceRecord trace) { + println "I completed a task! It's name is '${handler.task.name}'" + } + + @Override + void onProcessCached(TaskHandler handler, TraceRecord trace) { + println "I found a task in the cache! It's name is '${handler.task.name}'" + } + + @Override + void onFilePublish(Path destination, Path source) { + println "I published a file! It's located at ${path.toUriString()}" + } + + @Override + void onFlowError(TaskHandler handler, TraceRecord trace) { + println "Uh oh, something went wrong..." + } + + @Override + void onFlowComplete() { + println 'All done!' + } +} +``` + +You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: + +```nextflow +myplugin.enabled = true +``` + +See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. + +(dev-plugins-env-vars)= + +## Environment variables + +The following environment variables are available when developing and testing plugins: + +`NXF_PLUGINS_MODE` +: The plugin execution mode. Either `prod` for production or `dev` for development. + +`NXF_PLUGINS_DIR` +: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production and `./plugins` in development). + +`NXF_PLUGINS_DEFAULT` +: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: true). + + +`NXF_PLUGINS_DEV` +: Comma-separated list of development plugin root directories. + +`NXF_PLUGINS_TEST_REPOSITORY` +: :::{versionadded} 23.04.0. + ::: +: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. + +: The URI should refer to a plugin repository JSON file or a specific plugin JSON meta file. In the latter case, it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. For example: + + ```bash + # custom plugin repository at https://github.com/my-org/plugins + export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" + + # custom plugin release + export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" + + nextflow run main.nf -plugins nf-hello + ``` + +: This variable is useful for testing a plugin release before publishing it to the main registry. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md new file mode 100644 index 0000000000..470b8159a4 --- /dev/null +++ b/docs/plugins/plugins.md @@ -0,0 +1,68 @@ +(plugins-page)= + +# Overview + +## What are plugins + +Nextflow plugins are extensions that enhance the functionality of Nextflow. They allow users to add new capabilities and integrate with external services without bloating their pipeline code or modifying Nextflow itself. + +There are two types of Nextflow plugins: core plugins and third-party plugins. The main features of each plugin type are described below. + +

Core plugins

+ +Core plugins do not require configuration. The latest versions of core plugins are automatically installed when a Nextflow pipeline requests them. Core plugins include: + +* `nf-amazon`: Support for Amazon Web Services. +* `nf-azure`: Support for Microsoft Azure. +* `nf-cloudcache`: Support for the cloud cache. +* `nf-console`: Implementation of the Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). +* `nf-google`: Support for Google Cloud. +* `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/). +* `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/). + +Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. + +:::{note} +The automatic retrieval of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-vars` for more information. +::: + +

Third-party plugins

+ +Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: + +```groovy +plugins { + id 'nf-hello@0.5.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0 +``` + +See {ref}`using-plugins-page` for more information. + +## Nextflow plugin registry + +:::{versionadded} 25.04.0 +::: + +:::{note} +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. +::: + +The Nextflow plugin registry is a central repository for publishing and discovering Nextflow plugins. Nextflow is able to use the plugin registry to automatically find and download plugins at runtime. + +## Versioning + +Nextflow plugins are free to use any versioning convention. Many plugins, including all core plugins, use [Semantic Versioning](https://semver.org/), which helps developers communicate the kind of changes in a release. + +Semantic versions have the form MAJOR.MINOR.PATCH (e.g., 0.5.0): + +* MAJOR: Increment for backward-incompatible changes. +* MINOR: Increment for backward-compatible feature additions. +* PATCH: Increment for backward-compatible bug fixes. + +Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md new file mode 100644 index 0000000000..4e21292cc7 --- /dev/null +++ b/docs/plugins/using-plugins.md @@ -0,0 +1,79 @@ +(using-plugins-page)= + +# Using plugins + +Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin. + +(using-plugins-identifiers)= + +## Identifiers + +A plugin identifier consists of the plugin name and version, separated by an `@` symbol: + +``` +nf-hello@0.5.0 +``` + +The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin. + +:::{note} +Plugin versions are required for {ref}`offline usage `. +::: + +:::{versionadded} 25.02.0-edge +::: + +The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. + +:::{tip} +It is recommended to pin the major and minor version of each plugin that you use, in order to minimize the risk of breaking changes while allowing patch updates to be used automatically. +::: + +(using-plugins-config)= + +## Configuration + +Plugins can be configured via Nextflow configuration files or at runtime. + +To configure a plugin via configuration files, use the `plugins` block. For example: + +```nextflow +plugins { + id 'nf-hello@0.5.0' + id 'nf-amazon@2.9.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +``` + +:::{note} +Plugin declarations in Nextflow configuration files are ignored when specifying plugins via the `-plugins` option. +::: + +## Caching + +When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). + +(using-plugins-offline)= + +## Offline usage + +When running Nextflow in an offline environment, any required plugins must be downloaded and moved into the offline environment prior to any runs. + +To use Nextflow plugins in an offline environment: + +1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information. + +2. Run `nextflow plugin install @` for each required plugin to download it. Alternatively, run the pipeline once, which will automatically download all plugins required by the pipeline. + +3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. + +4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information. + + :::{warning} + Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information. + ::: diff --git a/docs/process.md b/docs/process.md index 78e9b35784..4eeb060a96 100644 --- a/docs/process.md +++ b/docs/process.md @@ -1164,7 +1164,7 @@ While this option can be used with any process output, it cannot be applied to i ## When -:::{node} +:::{note} As a best practice, conditional logic should be implemented in the calling workflow (e.g. using an `if` statement or {ref}`operator-filter` operator) instead of the process definition. ::: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 5a4e717f52..b9033e7a0a 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -957,7 +957,11 @@ Manage plugins and run plugin-specific commands. $ nextflow plugin [options] ``` -The `plugin` command provides several subcommands for managing and using plugins: +`create` + +: :::{versionadded} 25.04.0 + ::: +: Create a plugin scaffold using the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. `install ` diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index 6aad4b3a48..625283b6bb 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -159,7 +159,7 @@ The following environment variables control the configuration of the Nextflow ru `NXF_PLUGINS_TEST_REPOSITORY` : :::{versionadded} 23.04.0 ::: -: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. See {ref}`testing-plugins` for more information. +: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. `NXF_PUBLISH_FAIL_ON_ERROR` : :::{versionadded} 24.04.3 diff --git a/docs/reference/stdlib-groovy.md b/docs/reference/stdlib-groovy.md index 2d0bb83c8a..add202a445 100644 --- a/docs/reference/stdlib-groovy.md +++ b/docs/reference/stdlib-groovy.md @@ -21,5 +21,5 @@ println groovy.json.JsonOutput.toJson(vals) ``` :::{note} -The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. +The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. ::: diff --git a/docs/strict-syntax.md b/docs/strict-syntax.md index a546ac50c4..7f79344874 100644 --- a/docs/strict-syntax.md +++ b/docs/strict-syntax.md @@ -595,4 +595,4 @@ There are two ways to preserve Groovy code: Any Groovy code can be moved into the `lib` directory, which supports the full Groovy language. This approach is useful for temporarily preserving some Groovy code until it can be updated later and incorporated into a Nextflow script. See {ref}`lib-directory` documentation for more information. -For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`plugins-dev-page` for more information on how to develop plugins. +For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`dev-plugins-page` for more information on how to develop plugins.