From c487aaa86196402bd83abf237b33b38736391ba8 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Thu, 24 Apr 2025 19:57:55 +1200 Subject: [PATCH 01/19] Update plugins docs Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 134 ++++++++ docs/index.md | 13 + docs/migrating-gradle-plugin.md | 134 ++++++++ docs/plugins/developing-plugins.md | 489 +++++++++++++++++++++++++++++ docs/plugins/example-nf-hello.md | 52 +++ docs/plugins/plugins.md | 67 ++++ docs/plugins/using-plugins.md | 78 +++++ 7 files changed, 967 insertions(+) create mode 100644 docs/gradle-plugin.md create mode 100644 docs/migrating-gradle-plugin.md create mode 100644 docs/plugins/developing-plugins.md create mode 100644 docs/plugins/example-nf-hello.md create mode 100644 docs/plugins/plugins.md create mode 100644 docs/plugins/using-plugins.md diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md new file mode 100644 index 0000000000..31f341f0ee --- /dev/null +++ b/docs/gradle-plugin.md @@ -0,0 +1,134 @@ +# Using the Gradle plugin + +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. + +:::{note} +Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +## Creating a plugin + +The nf-hello plugin uses the Gradle plugin and is a valuable starting point for developers. + +To create a Nextflow plugin with the Gradle plugin: + +1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. + * See Example: nf-hello for more information about the nf-hello plugin. +2. Rename the forked `nf-hello` directory with your plugin name. +3. Replace the contents of `settings.gradle` with the following: + + ```groovy + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. Replace the contents of `build.gradle` with the following: + + ```groovy + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + github { + repository = '' + userName = project.findProperty('github_username') + authToken = project.findProperty('github_access_token') + email = project.findProperty('github_commit_email') + indexUrl = '' + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (optional) your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: the minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. + - `PROVIDER`: your name or organization—for example, `nextflow`. + - `CLASS_NAME`: your plugin class name—for example, `nextflow.hello.HelloPlugin`. + - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. + - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. + - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). +5. Develop your plugin extension points: + - See Extension points for descriptions and examples. +6. In the plugin root directory, run `make assemble`. + + +## Installing a plugin + +Plugins can be installed locally without being packaged, uploaded, and published. + +To install a plugin locally: + +1. In the plugin root directory, run `make install`. + + :::{note} + Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. + ::: + +2. Configure your plugin. + * See Using plugins for more information. +3. Run your pipeline: + + ```bash + nextflow run main.nf + ``` + +## Unit testing a plugin + +Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. + +To run unit tests: + +1. Develop your unit tests. + * See HelloDslTest.groovy in the nf-hello plugin for unit test examples. +2. In the plugin root directory, run `make test`. + + +### Packaging, uploading, and publishing a plugin + +The Gradle plugin for Nextflow plugins simplifies publishing your plugin. + +To package, upload, and publish your plugin: + +1. Fork the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins). +2. In the plugin root directory, open `build.gradle` and ensure that: + * `github.repository` matches the plugin repository. + * `github.indexUrl` matches your fork of the plugins index repository. +3. Create a file named `$HOME/.gradle/gradle.properties` and add the following: + + ```bash + github_username= + github_access_token= + github_commit_email= + ``` + + Replace the following: + * `GITHUB_USERNAME`: your GitHub username granting access to the plugin repository. + * `GITHUB_ACCESS_TOKEN`: your GitHub access token with permission to upload and commit changes to the plugin repository. + * `GITHUB_EMAIL`: your email address associated with your GitHub account. +4. Run `make release`. +5. Create a pull request against the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins) from your fork. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index fe8098efc7..7943bd88b7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -149,11 +149,24 @@ developer/packages developer/plugins ``` +```{toctree} +:hidden: +:caption: Plugins +:maxdepth: 1 + +plugins/plugins +plugins/using-plugins +plugins/developing-plugins +plugins/example-nf-hello +``` + ```{toctree} :hidden: :caption: Guides :maxdepth: 1 +gradle-plugin +migrating-gradle-plugin updating-spot-retries metrics flux diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md new file mode 100644 index 0000000000..b424ff4233 --- /dev/null +++ b/docs/migrating-gradle-plugin.md @@ -0,0 +1,134 @@ +# Migrating to the Gradle plugin for Nextflow plugins + +This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. + + +## Improvements to the plugin framework + +The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. + +### Gradle plugin for Nextflow plugins + +The Gradle plugin for Nextflow plugins simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. + +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. + +### Nextflow plugin registry + +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. + +Developers can upload plugins to the plugin registry. The Gradle plugin for Nextflow plugins supports this process with Gradle tasks that package, upload, and publish plugins. + +## Impact on users and developers + +The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. + +### Plugin Users + +If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. + +### Plugin developers + +Developers are encouraged to migrate to the Gradle plugin for Nextflow plugins and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. + +To migrate an existing Nextflow plugin: + +1. Remove the following files and folders: + - `buildSrc/` + - `nextflow.config` + - `launch.sh` + - `plugins/build.gradle` +2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. \ + + :::{note} + Plugin sources should be in `src/main/groovy` or `src/main/java`. + ::: + +3. Replace the contents of `settings.gradle` with the following: + + ``` + rootProject.name = '' + ``` + + Replace `PLUGIN_NAME` with your plugin name. + +4. In the project root, create a new `build.gradle` file with the following configuration: + + ``` + // Plugins + plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + } + + // Dependencies (optional) + dependencies { + + } + + // Plugin version + version = '' + + nextflowPlugin { + // Minimum Nextflow version + nextflowVersion = '' + + // Plugin metadata + provider = '' + className = '' + extensionPoints = [ + '' + ] + + publishing { + github { + repository = '' + userName = project.findProperty('github_username') + authToken = project.findProperty('github_access_token') + email = project.findProperty('github_commit_email') + indexUrl = '' + } + } + } + ``` + + Replace the following: + + - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. + - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. + - `PROVIDER`: Your name or organization—for example, `nextflow`. + - `CLASS_NAME`: Your plugin class name—for example, `nextflow.hello.HelloPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. + - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. + - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). + +5. Replace the contents of `Makefile` with the following: + + ``` + # Build the plugin + assemble: + ./gradlew assemble + + clean: + rm -rf .nextflow* + rm -rf work + rm -rf build + ./gradlew clean + + # Run plugin unit tests + test: + ./gradlew test + + # Install the plugin into local nextflow plugins dir + install: + ./gradlew install + + # Publish the plugin + release: + ./gradlew releasePlugin + ``` + +6. Update `README.md` with information about the structure of your plugin. +7. In the plugin root directory, run `make assemble`. + +The Gradle plugin for Nextflow plugins also supports publishing plugins. See Packaging, uploading, and publishing for more information. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md new file mode 100644 index 0000000000..07b3c3fe2a --- /dev/null +++ b/docs/plugins/developing-plugins.md @@ -0,0 +1,489 @@ +(developing-plugins-page)= + +# Developing plugins + +The Nextflow plugin framework streamlines plugin development by providing the structure and tools needed to extend Nextflow functionality. The Gradle plugin for Nextflow plugins simplifies development by configuring default Nextflow dependencies and incorporates custom Gradle tasks that streamline building and publishing plugins. + +:::{note} +Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +(developing-plugins-framework)= + +## Framework + +Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. + +(developing-plugins-gradle)= + +## Gradle plugin for Nextflow plugins + +[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. + +(developing-plugins-structure)= + +### Structure + +Plugins built with the Gradle plugin for Nextflow plugins follow a standard project layout. This structure includes source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. + +A typical plugin built with the Gradle plugin for Nextflow plugins has the following structure: + +``` +. +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ └── ... +├── gradlew +├── settings.gradle +└── src + ├── main + │ └── ... + └── test + └── ... +``` + +This structure contains the following key files and folders: + +- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. +- `src/main/`: The main source directory containing the plugin's implementation and resources.​ +- `src/test/`: The main source directory containing the plugin's unit testing. +- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ +- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ +- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ +- `build.gradle`: The primary build script for Gradle. +- `gradlew`: A script for executing the Gradle Wrapper. +- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. + +See {ref}`nf-hello-page` for an example of a plugin built using this structure. + +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. See nf-hello for an example of a plugin with the old structure. +::: + +(developing-plugins-make)= + +### make commands + +The Gradle plugin for Nextflow plugins defines tasks that can be executed with `./gradlew`. For example: + +```bash +./gradlew assemble +``` + +For convenience, the more important tasks are wrapped in a makefile and can be executed with the `make` command. For example: + +```bash +make assemble +``` + +The following `make` commands are available: + +`assemble` +: Compiles the Nextflow plugin code and assembles it into a zip file. See Creating a plugin for more information. + +`test` +: Runs plugin unit tests. See Unit testing for more information. + +`install` +: Installs the plugin into the local nextflow plugins directory. See Running locally for more information. + +`release` +: Publishes the plugin. See Packaging, uploading, and publishing for more information. + +(developing-plugins-versioning)= + +### Versioning + +The Gradle plugin is versioned and published to a [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: + +```nextflow +plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' +} +``` + +See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. + +(developing-plugins-extension)= + +## Extension points + +Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how they can be used. + +### Commands + +Plugins can define custom CLI commands that are executable with the `nextflow plugin` command. + +To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entry point (the class that extends `BasePlugin`). Alternatively, implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement the `getCommands()` and `exec()` methods. For example: + +```groovy +import nextflow.cli.PluginAbstractExec +import nextflow.plugin.BasePlugin + +class MyPlugin extends BasePlugin implements PluginAbstractExec { + @Override + List getCommands() { + [ 'hello' ] + } + + @Override + int exec(String cmd, List args) { + if( cmd == 'hello' ) { + println "Hello! You gave me these arguments: ${args.join(' ')}" + return 0 + } + else { + System.err.println "Invalid command: ${cmd}" + return 1 + } + } +} +``` + +The command can be run using the `nextflow plugin` command: + +``` +nextflow plugin my-plugin:hello --foo --bar +``` + +See the {ref}`cli-plugin` for usage information. + +### Configuration + +Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option safely. If the option isn’t defined, it will return null. + +A common practice is to use a custom config scope to define any configuration for your plugin. For example: + +```groovy +import nextflow.Session +import nextflow.trace.TraceObserver + +class MyObserver implements TraceObserver { + + @Override + void onFlowCreate(Session session) { + final message = session.config.navigate('myplugin.createMessage') + println message + } +} +``` + +This option can then be set in your configuration file: + +```groovy +// dot syntax +myplugin.createMessage = "I'm alive!" + +// block syntax +myplugin { + createMessage = "I'm alive!" +} +``` + +:::{versionadded} 25.02.0-edge +::: + +Plugins can declare their configuration options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation. For example: + +```groovy +import nextflow.config.schema.ConfigOption +import nextflow.config.schema.ConfigScope +import nextflow.config.schema.ScopeName +import nextflow.script.dsl.Description + +@ScopeName('myplugin') +@Description(''' + The `myplugin` scope allows you to configure the `nf-myplugin` plugin. +''') +class MyPluginConfig implements ConfigScope { + + MyPluginConfig(Map opts) { + this.createMessage = opts.createMessage + } + + @ConfigOption + @Description('Message to print to standard output when a run is initialized.') + String createMessage +} +``` + +This approach is not required to support plugin config options. However, it allows Nextflow to recognize plugin definitions when validating the configuration. + +### Executors + +Plugins can define custom executors that can be used with the `executor` process directive. + +To implement an executor, create a class in your plugin that extends the [Executor](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: + +```groovy +import nextflow.executor.Executor +import nextflow.util.ServiceName +import org.pf4j.ExtensionPoint + +@ServiceName('my-executor') +class MyExecutor extends Executor implements ExtensionPoint { + + // ... + +} +``` + +You can then use this executor in your pipeline: + +```groovy +process foo { + executor 'my-executor' + + // ... +} +``` + +:::{tip} +See the source code of Nextflow's built-in executors for examples of how to implement various executor components. +::: + + +### Filesystems + +Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see Remote files. + +To implement a custom filesystem, create a class in your plugin that extends [FileSystemProvider](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: + +```groovy +import java.nio.file.spi.FileSystemProvider + +class MyFileSystemProvider extends FileSystemProvider { + + @Override + String getScheme() { + return 'myfs' + } + + // ... +} +``` + +You can then use this filesystem in your pipeline: + +```nextflow +input = file('myfs://') +``` + +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for examples of custom filesystems. + +:::{tip} +Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3. +::: + +### Functions + +:::{versionadded} 22.09.0-edge +::: + +Plugins can define custom functions that can be included in Nextflow pipelines. + +To implement a custom function, create a plugin class that extends the `PluginExtensionPoint` class and implement your function with the `Function` annotation. For example: + +```groovy +import nextflow.Session +import nextflow.plugin.extension.Function +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Function + String reverseString(String origin) { + origin.reverse() + } +} +``` + +You can then add this function to your pipeline: + +``` +include { reverseString } from 'plugin/my-plugin' + +channel.of( reverseString('hi') ) +``` + +Alternatively, you can use an alias: + +```nextflow +include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' +``` + +### Operators + +:::{versionadded} 22.04.0 +::: + +Plugins can define custom channel factories and operators that can then be included in pipelines. + +To implement a custom channel factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class and implement your function with the `Factory` or `Operator` annotation. For example: + +```groovy +import groovyx.gpars.dataflow.DataflowReadChannel +import groovyx.gpars.dataflow.DataflowWriteChannel +import nextflow.Session +import nextflow.plugin.extension.Factory +import nextflow.plugin.extension.Operator +import nextflow.plugin.extension.PluginExtensionPoint + +class MyExtension extends PluginExtensionPoint { + + @Override + void init(Session session) {} + + @Factory + DataflowWriteChannel fromQuery(Map opts, String query) { + // ... + } + + @Operator + DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { + // ... + } + +} +``` + +You can then use the custom channel factories or operators in your pipeline: + +```nextflow +include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' + +def sql = 'select * from FOO' +channel + .fromTable(sql, db: 'test', emitColumns: true) + .sqlInsert(into: 'BAR', columns: 'id', db: 'test') +``` + +:::{note} +The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +::: + +### Process directives + +Plugins that implement a custom executor will likely need to access process directives that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. + +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the ext directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. + +For example, a custom executor can use existing process directives and a custom setting through the `ext` directive: + +```groovy +class MyExecutor extends Executor { + + @Override + TaskHandler createTaskHandler(TaskRun task) { + final cpus = task.config.cpus + final memory = task.config.memory + final myOption = task.config.ext.myOption + + println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" + + // ... + } + + // ... + +} +``` + +### Trace observers + +A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. + +Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example: + +```groovy +import java.nio.file.Path + +import nextflow.processor.TaskHandler +import nextflow.trace.TraceObserver +import nextflow.trace.TraceRecord + +class MyObserver implements TraceObserver { + + @Override + void onFlowBegin() { + println "Okay, let's begin!" + } + + @Override + void onProcessComplete(TaskHandler handler, TraceRecord trace) { + println "I completed a task! It's name is '${handler.task.name}'" + } + + @Override + void onProcessCached(TaskHandler handler, TraceRecord trace) { + println "I found a task in the cache! It's name is '${handler.task.name}'" + } + + @Override + void onFilePublish(Path destination, Path source) { + println "I published a file! It's located at ${path.toUriString()}" + } + + @Override + void onFlowError(TaskHandler handler, TraceRecord trace) { + println "Uh oh, something went wrong..." + } + + @Override + void onFlowComplete() { + println 'All done!' + } +} +``` + +You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: + +```nextflow +myplugin.enabled = true +``` + +See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. + + +## Environment variables + +The following environment variables are available when developing and testing plugins: + +`NXF_PLUGINS_MODE` +: The plugin execution mode. Either `prod` for production or `dev` for development. + +`NXF_PLUGINS_DIR` +: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production and `./plugins` in development). + +`NXF_PLUGINS_DEFAULT` +: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: true). + + +`NXF_PLUGINS_DEV` +: Comma-separated list of development plugin root directories. + +`NXF_PLUGINS_TEST_REPOSITORY` +: :::{versionadded} 23.04.0. + ::: +: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. + +: The URI should refer to a plugin repository JSON file or a specific plugin JSON meta file. In the latter case, it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. For example: + + ```bash + # custom plugin repository at https://github.com/my-org/plugins + export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" + + # custom plugin release + export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" + + nextflow run main.nf -plugins nf-hello + ``` + +: This variable is useful for testing a plugin release before publishing it to the main registry. diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md new file mode 100644 index 0000000000..0b96fb4d07 --- /dev/null +++ b/docs/plugins/example-nf-hello.md @@ -0,0 +1,52 @@ +(nf-hello-page)= + +# Example: nf-hello + +`nf-hello` is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. + +The nf-hello plugin has the following structure: + +``` +nf-hello +├── COPYING +├── Makefile +├── README.md +├── build.gradle +├── gradle +│ └── wrapper +│ ├── gradle-wrapper.jar +│ └── gradle-wrapper.properties +├── gradlew +├── settings.gradle +└── src + ├── main + │ └── groovy + │ └── nextflow + │ └── hello + │ ├── HelloConfig.groovy + │ ├── HelloExtension.groovy + │ ├── HelloFactory.groovy + │ ├── HelloObserver.groovy + │ └── HelloPlugin.groovy + └── test + └── groovy + └── nextflow + └── hello + ├── HelloDslTest.groovy + └── HelloFactoryTest.groovy +``` + +It includes examples of different plugin extensions: + +- A custom trace observer that prints a message when the workflow starts and when the workflow completes. +- A custom channel factory called reverse. +- A custom operator called goodbye. +- A custom function called randomString. + +It also includes several classes that demonstrate different plugin functionality: + +- `HelloConfig`: An example of how to handle options from the Nextflow configuration. +- `HelloExtension`: An example of how to create custom channel factories, operators, and functions that can be included in pipeline scripts. +- `HelloFactory`: An example of a workflow event with custom behavior. +- `HelloObserver`: An example of a workflow event with custom behavior. +- `HelloPlugin`: An example of a plugin entry point. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md new file mode 100644 index 0000000000..259d1e9e0c --- /dev/null +++ b/docs/plugins/plugins.md @@ -0,0 +1,67 @@ +(plugins-page)= + +# Overview + +## What are plugins + +Nextflow plugins are extensions that enhance the functionality of the Nextflow workflow framework. They allow users to add new capabilities and integrate with external services without modifying core Nextflow code. + +There are two types of Nextflow plugins: + +* Core plugins +* Third-party plugins + +The main features of each plugin type are described below. + +

Core plugins

+ +Core plugins do not require configuration. The latest versions of core plugins are automatically installed when a Nextflow pipeline requests them. Core plugins include: + +* `nf-amazon`: Support for Amazon Web Services. +* `nf-azure`: Support for Microsoft Azure. +* `nf-cloudcache`: Support for the cloud cache. +* `nf-console`: Implement Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). +* `nf-google`: Support for Google Cloud. +* `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/). +* `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/). + +Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. + +:::{note} +The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See Environment variables for more information. +::: + +

Third-party plugins

+ +Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: + +``` +plugins { + id 'nf-hello@0.5.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +``` +nextflow run main.nf -plugins nf-hello@0.5.0 +``` + +See {ref}`using-plugins-page` for more information. + +## Nextflow plugin registry + +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. Developers can upload plugins to the registry, with built-in support from the Gradle plugin for Nextflow plugins. + +## Versioning + +Nextflow plugins follow the Semantic Versioning format: MAJOR.MINOR.PATCH (e.g., 0.5.0). This helps developers communicate the nature of changes in a release. + +Version components: + +* MAJOR: Increment for incompatible changes. +* MINOR: Increment for backward-compatible feature additions. +* PATCH: Increment for backward-compatible bug fixes. + +Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. +See Semantic Versioning for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md new file mode 100644 index 0000000000..8aef78fc7e --- /dev/null +++ b/docs/plugins/using-plugins.md @@ -0,0 +1,78 @@ +# Using plugins + +Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin. + +## Identifiers + +A plugin identifier consists of the plugin name and version, separated by an `@` symbol: + +```console +nf-hello@0.5.0 +``` + +The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin. + +:::{note} +Plugin versions are required to run plugins in an Offline environment. +::: + +:::{versionadded} 25.02.0-edge. +::: + +The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. + +:::{tip} +It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. +::: + +## Configuration + +Plugins can be configured via nextflow configuration files or at runtime. + +To configure a plugin via configuration files, use the `plugins` block. For example: + +```nextflow +plugins { + id 'nf-hello@0.5.0' +} +``` + +To configure plugins at runtime, use the `-plugins` option. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0 +``` + +To configure multiple plugins at runtime, use the `-plugins` option and a comma-separated list. For example: + +```bash +nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +``` + +:::{note} +Plugin declarations in nextflow configuration files are ignored when specifying plugins via the `-plugins` option. +::: + +## Caching + +When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. + +## Offline usage + +Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. + +To use Nextflow plugins in an offline environment: + +1. Install a self-contained version of Nextflow in an environment with an internet connection. See Standalone distribution for more information. +2. Run `nextflow plugin install @` to download the plugins. + + :::{tip} + Running a pipeline in an environment with an internet connection will also download the plugins used by that pipeline. + ::: + +3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. +4. Specify each plugin and its version in Nextflow configuration files or at runtime. See Configuration for more information. + + :::{warning} + Nextflow will attempt to download newer versions of plugins if their versions are not set. See Plugin identifiers for more information. + ::: \ No newline at end of file From a0463318c5f7ef59e104a3b9b9357840c024ccc4 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 28 Apr 2025 09:36:15 +1200 Subject: [PATCH 02/19] Add run command example Signed-off-by: Christopher Hakkaart --- docs/plugins/example-nf-hello.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 0b96fb4d07..689b5f0a69 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -50,3 +50,9 @@ It also includes several classes that demonstrate different plugin functionality - `HelloFactory`: An example of a workflow event with custom behavior. - `HelloObserver`: An example of a workflow event with custom behavior. - `HelloPlugin`: An example of a plugin entry point. + +The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: + +```bash +nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +``` From 989af211b9e8c0752075845ce394f4170ddcf1cd Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 28 Apr 2025 13:40:43 +1200 Subject: [PATCH 03/19] Update links Signed-off-by: Christopher Hakkaart --- docs/developer/plugins.md | 541 ----------------------------- docs/gradle-plugin.md | 24 +- docs/index.md | 2 - docs/migrating-gradle-plugin.md | 6 +- docs/plugins.md | 63 ---- docs/plugins/developing-plugins.md | 37 +- docs/plugins/example-nf-hello.md | 4 +- docs/plugins/plugins.md | 4 +- docs/plugins/using-plugins.md | 16 +- docs/reference/env-vars.md | 2 +- docs/strict-syntax.md | 2 +- 11 files changed, 54 insertions(+), 647 deletions(-) delete mode 100644 docs/developer/plugins.md delete mode 100644 docs/plugins.md diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md deleted file mode 100644 index 7ea5a779e5..0000000000 --- a/docs/developer/plugins.md +++ /dev/null @@ -1,541 +0,0 @@ -(plugins-dev-page)= - -# Plugins - -This page describes how to create, test, and publish third-party plugins. - -## Plugin structure - -The best way to get started with your own plugin is to refer to the [nf-hello](https://github.com/nextflow-io/nf-hello) repository. This repository provides a minimal plugin implementation with several examples of different extension points and instructions for building, testing, and publishing. - -Plugins can be written in Java or Groovy. - -The minimal dependencies are as follows: - -```groovy -dependencies { - compileOnly project(':nextflow') - compileOnly 'org.slf4j:slf4j-api:1.7.10' - compileOnly 'org.pf4j:pf4j:3.4.1' - - testImplementation project(':nextflow') - testImplementation "org.codehaus.groovy:groovy:4.0.24" - testImplementation "org.codehaus.groovy:groovy-nio:4.0.23" -} -``` - -The plugin subproject directory name must begin with the prefix `nf-` and must include a file named `src/resources/META-INF/MANIFEST.MF` which contains the plugin metadata. The manifest content looks like the following: - -``` -Manifest-Version: 1.0 -Plugin-Class: the.plugin.ClassName -Plugin-Id: the-plugin-id -Plugin-Provider: Your Name or Organization -Plugin-Version: 0.0.0 -``` - -## Extension points - -Nextflow's plugin system exposes a variety of extension points for plugins. This section describes how to use these extension points when writing a plugin, as well as how they are used in a pipeline. - -:::{note} -If you would like to implement something in a plugin that isn't covered by any of the following sections, feel free to create an issue on GitHub and describe your use case. In general, any class in the Nextflow codebase that implements `ExtensionPoint` can be extended by a plugin, and existing plugins are a great source of examples when writing new plugins. -::: - -:::{note} -Plugin extension points must be added to `extensions.idx` in the plugin repository to make them discoverable. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) plugin for an example. -::: - -### Commands - -Plugins can define custom CLI commands that can be executed with the `nextflow plugin` command. - -To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entrypoint (the class that extends `BasePlugin`). Alternatively, you can implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement two methods, `getCommands()` and `exec()`: - -```groovy -import nextflow.cli.PluginAbstractExec -import nextflow.plugin.BasePlugin - -class MyPlugin extends BasePlugin implements PluginAbstractExec { - @Override - List getCommands() { - [ 'hello' ] - } - - @Override - int exec(String cmd, List args) { - if( cmd == 'hello' ) { - println "Hello! You gave me these arguments: ${args.join(' ')}" - return 0 - } - else { - System.err.println "Invalid command: ${cmd}" - return 1 - } - } -} -``` - -You can then execute this command using the `nextflow plugin` command: - -```bash -nextflow plugin my-plugin:hello --foo --bar -``` - -See the {ref}`cli-plugin` CLI command for usage information. - -### Configuration - -Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option in a safe manner -- if the option isn't defined, it will return `null`. A common practice is to define any configuration for your plugin in a custom config scope. - -For example, you can query a config option in a trace observer hook: - -```groovy -import nextflow.Session -import nextflow.trace.TraceObserver - -class MyObserver implements TraceObserver { - - @Override - void onFlowCreate(Session session) { - final message = session.config.navigate('myplugin.createMessage') - println message - } -} -``` - -You can then set this option in your config file: - -```groovy -// dot syntax -myplugin.createMessage = "I'm alive!" - -// block syntax -myplugin { - createMessage = "I'm alive!" -} -``` - -:::{versionadded} 25.02.0-edge -::: - -Plugins can declare their config options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation: - -```groovy -import nextflow.config.schema.ConfigOption -import nextflow.config.schema.ConfigScope -import nextflow.config.schema.ScopeName -import nextflow.script.dsl.Description - -@ScopeName('myplugin') -@Description(''' - The `myplugin` scope allows you to configure the `nf-myplugin` plugin. -''') -class MyPluginConfig implements ConfigScope { - - MyPluginConfig(Map opts) { - this.createMessage = opts.createMessage - } - - @ConfigOption - @Description('Message to print to standard output when a run is initialized.') - String createMessage -} -``` - -While this approach is not required to support plugin config options, it allows Nextflow to recognize plugin definitions when validating the config. - -### Executors - -Plugins can define custom executors that can then be used with the `executor` process directive. - -To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor: - -```groovy -import nextflow.executor.Executor -import nextflow.util.ServiceName -import org.pf4j.ExtensionPoint - -@ServiceName('my-executor') -class MyExecutor extends Executor implements ExtensionPoint { - - // ... - -} -``` - -You can then use this executor in your pipeline: - -```nextflow -process foo { - executor 'my-executor' - - // ... -} -``` - -:::{tip} -Refer to the source code of Nextflow's built-in executors to see how to implement the various components of an executor. You might be able to implement most of your executor by simply reusing existing code. -::: - -### Filesystems - -Plugins can define custom filesystems that can be used by Nextflow to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. - -To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem: - -```groovy -import java.nio.file.spi.FileSystemProvider - -class MyFileSystemProvider extends FileSystemProvider { - - @Override - String getScheme() { - return 'myfs' - } - - // ... -} -``` - -You can then use this filesystem in your pipeline: - -```nextflow -input = file('myfs://path/to/input/file.txt') -``` - -See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information. Refer to the `nf-https` module (`XFileSystemProvider`) or `nf-amazon` plugin (`S3FileSystemProvider`) in the Nextflow source code for examples of custom filesystems. - -:::{tip} -Custom filesystems are an advanced type of plugin extension. Before creating a new filesystem, make sure that your use case can't already be supported through an existing filesystem such as HTTP or S3. -::: - -### Functions - -:::{versionadded} 22.09.0-edge -::: - -Plugins can define custom functions, which can then be included in Nextflow pipelines. - -To implement a custom function, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Function` annotation: - -```groovy -import nextflow.Session -import nextflow.plugin.extension.Function -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Function - String reverseString(String origin) { - origin.reverse() - } - -} -``` - -You can then use this function in your pipeline: - -```nextflow -include { reverseString } from 'plugin/my-plugin' - -channel.of( reverseString('hi') ) -``` - -You can also use an alias: - -```nextflow -include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' -``` - -### Operators - -:::{versionadded} 22.04.0 -::: - -Plugins can define custom channel factories and operators, which can then be included into Nextflow pipelines. - -To implement a custom factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class, and implement your function with the `Factory` or `Operator` annotation: - -```groovy -import groovyx.gpars.dataflow.DataflowReadChannel -import groovyx.gpars.dataflow.DataflowWriteChannel -import nextflow.Session -import nextflow.plugin.extension.Factory -import nextflow.plugin.extension.Operator -import nextflow.plugin.extension.PluginExtensionPoint - -class MyExtension extends PluginExtensionPoint { - - @Override - void init(Session session) {} - - @Factory - DataflowWriteChannel fromQuery(Map opts, String query) { - // ... - } - - @Operator - DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) { - // ... - } - -} -``` - -You can then use them in your pipeline: - -```nextflow -include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' - -def sql = 'select * from FOO' -channel - .fromTable(sql, db: 'test', emitColumns: true) - .sqlInsert(into: 'BAR', columns: 'id', db: 'test') -``` - -The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory -is included under the alias `fromTable`. - -### Process directives - -Plugins that implement a [custom executor](#executors) will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task's configuration. As a best practice, custom executors should try to support all process directives that have executor-specific behavior and are relevant to your executor. - -Nextflow does not provide the ability to define custom process directives in a plugin. Instead, you can use the {ref}`process-ext` directive to provide custom process settings to your executor. Try to use specific names that are not likely to conflict with other plugins or existing pipelines. - -For example, a custom executor can use existing process directives as well as a custom setting through the `ext` directive: - -```groovy -class MyExecutor extends Executor { - - @Override - TaskHandler createTaskHandler(TaskRun task) { - final cpus = task.config.cpus - final memory = task.config.memory - final myOption = task.config.ext.myOption - - println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}" - - // ... - } - - // ... - -} -``` - -### Trace observers - -A *trace observer* in Nextflow is an entity that can listen and react to workflow events, such as when a workflow starts, a task completes, a file is published, etc. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. - -Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver`, and implement the `create()` method in your observer factory: - -```groovy -// MyObserverFactory.groovy -import nextflow.Session -import nextflow.trace.TraceObserver -import nextflow.trace.TraceObserverFactory - -class MyObserverFactory implements TraceObserverFactory { - - @Override - Collection create(Session session) { - final enabled = session.config.navigate('myplugin.enabled') - return enabled ? [ new MyObserver() ] : [] - } -} - -// MyObserver.groovy -import java.nio.file.Path - -import nextflow.processor.TaskHandler -import nextflow.trace.TraceObserver -import nextflow.trace.TraceRecord - -class MyObserver implements TraceObserver { - - @Override - void onFlowBegin() { - println "Okay, let's begin!" - } - - @Override - void onProcessComplete(TaskHandler handler, TraceRecord trace) { - println "I completed a task! It's name is '${handler.task.name}'" - } - - @Override - void onProcessCached(TaskHandler handler, TraceRecord trace) { - println "I found a task in the cache! It's name is '${handler.task.name}'" - } - - @Override - void onFilePublish(Path destination, Path source) { - println "I published a file! It's located at ${path.toUriString()}" - } - - @Override - void onFlowError(TaskHandler handler, TraceRecord trace) { - println "Uh oh, something went wrong..." - } - - @Override - void onFlowComplete() { - println 'All done!' - } -} -``` - -You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option: - -```groovy -myplugin.enabled = true -``` - -Refer to the `TraceObserver` [source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. - -## Publishing plugins - -Nextflow resolves plugins through a plugin registry, which stores metadata for each plugin version, including the publishing date, checksum, and download URL for the plugin binary. The default registry is located on GitHub at [nextflow-io/plugins](https://github.com/nextflow-io/plugins/), specifically [this file](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -To publish a plugin, you must create the plugin release, publish it to GitHub, and make a pull request against the main plugin registry with the metadata for your plugin release. - -A plugin release is a ZIP archive containing the compiled plugin classes, the required dependencies, and a JSON file with the plugin metadata. See the [`nf-hello`](https://github.com/nextflow-io/nf-hello) example to see how to create a plugin release. - -Here is an example meta file for a plugin release: - -```json -{ - "version": "0.2.0", - "url": "https://github.com/nextflow-io/nf-amazon/releases/download/0.2.0/nf-amazon-0.2.0.zip", - "date": "2020-10-12T10:05:44.28+02:00", - "sha512sum": "9e9e33695c1a7c051271..." -} -``` - -:::{note} -The plugin version should be a valid [semantic version](https://semver.org/). -::: - -(testing-plugins)= - -## Testing plugins - -Plugins must be declared in the `nextflow.config` file using the `plugins` scope, for example: - -```groovy -plugins { - id 'nf-amazon@0.2.0' -} -``` - -If a plugin is not locally available, Nextflow checks the repository index for the download URL, downloads and extracts the plugin archive, and installs the plugin into the directory specified by `NXF_PLUGINS_DIR` (default: `${NXF_HOME}/plugins`). - -Since each Nextflow run can have a different set of plugins (and versions), each run keeps a local plugins directory called `.nextflow/plr/` which links the exact set of plugins required for the given run. - -Additionally, the "default plugins" (defined in the Nextflow resources file `modules/nextflow/src/main/resources/META-INF/plugins-info.txt`) are always made available for use. To disable the use of default plugins, set the environment variable `NXF_PLUGINS_DEFAULT=false`. - -When running in development mode, the plugin system uses the `DevPluginClasspath` to load plugin classes from each plugin project build path, e.g. `plugins/nf-amazon/build/classes` and `plugins/nf-amazon/build/target/libs` (for deps libraries). - -## Environment variables - -The following environment variables are available when developing and testing plugins: - -`NXF_PLUGINS_MODE` -: The plugin execution mode, either `prod` for production or `dev` for development (see below for details). - -`NXF_PLUGINS_DIR` -: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production, `./plugins` in development). - -`NXF_PLUGINS_DEFAULT` -: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: `true`). - -`NXF_PLUGINS_DEV` -: Comma-separated list of development plugin root directories. - -`NXF_PLUGINS_TEST_REPOSITORY` -: :::{versionadded} 23.04.0 - ::: -: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry. -: The URI should refer to a plugins repository JSON file or a specific plugin JSON meta file. In the latter case it should match the pattern `https://host.name/some/path/-X.Y.Z-meta.json`. - -: For example: - - ```bash - # custom plugin repository at https://github.com/my-org/plugins - export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json" - - # custom plugin release - export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json" - - nextflow run -plugins nf-hello - ``` - -: This variable is useful for testing a plugin release before publishing it to the main registry. - -## Gradle Tasks - -The following build tasks are defined in the `build.gradle` of each core plugin as well as the `nf-hello` example plugin. - -### `makeZip` - -Create the plugin archive and JSON meta file in the subproject `build/libs` directory. - -```console -$ ls -l1 plugins/nf-tower/build/libs/ -nf-tower-0.1.0.jar -nf-tower-0.1.0.json -nf-tower-0.1.0.zip -``` - -### `copyPluginLibs` - -Copy plugin dependencies JAR files into the subproject `build/target/libs` directory. This is needed only when launching the plugin in development mode. - -### `copyPluginZip` - -Copy the plugin archive to the root project build directory, i.e. `build/plugins`. - -### `uploadPlugin` - -Upload the plugin archive and meta file to the corresponding GitHub repository. Options: - -`release` -: The plugin version, e.g. `1.0.1`. - -`repo` -: The GitHub repository name, e.g. `nf-amazon`. - -`owner` -: The GitHub owning organization, e.g. `nextflow-io`. - -`skipExisting` -: Do not upload a file if it already exists, i.e. checksum is the same (default: `true`). - -`dryRun` -: Execute the tasks without uploading file (default: `false`). - -`overwrite` -: Prevent to overwrite a remote file already existing (default: `false`). - -`userName` -: The user name used to authenticate GitHub API requests. - -`authToken` -: The personal token used to authenticate GitHub API requests. - -### `upload` - -Upload the plugin archive and meta file. - -### `publishIndex` - -Upload the plugins index to the repository hosted at [nextflow-io/plugins](https://github.com/nextflow-io/plugins), which makes them publicly accessible at [this URL](https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json). - -## Additional Resources - -* [PF4J](https://pf4j.org/) -* [Understanding Gradle: The Build Lifecycle](https://proandroiddev.com/understanding-gradle-the-build-lifecycle-5118c1da613f) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 31f341f0ee..2eb45aa150 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -1,3 +1,5 @@ +(gradle-plugin-page)= + # Using the Gradle plugin The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. @@ -6,14 +8,15 @@ The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: +(gradle-plugin-create)= + ## Creating a plugin -The nf-hello plugin uses the Gradle plugin and is a valuable starting point for developers. +The [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin uses the Gradle plugin and is a valuable starting point for developers. To create a Nextflow plugin with the Gradle plugin: -1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. - * See Example: nf-hello for more information about the nf-hello plugin. +1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. See {ref}`nf-hello-page` for more information. 2. Rename the forked `nf-hello` directory with your plugin name. 3. Replace the contents of `settings.gradle` with the following: @@ -72,10 +75,10 @@ To create a Nextflow plugin with the Gradle plugin: - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). -5. Develop your plugin extension points: - - See Extension points for descriptions and examples. +5. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. 6. In the plugin root directory, run `make assemble`. +(gradle-plugin-install)= ## Installing a plugin @@ -89,26 +92,27 @@ To install a plugin locally: Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. ::: -2. Configure your plugin. - * See Using plugins for more information. +2. Configure your plugin. See {ref}`using-plugins-page` for more information. 3. Run your pipeline: ```bash nextflow run main.nf ``` +(gradle-plugin-unit-test)= + ## Unit testing a plugin Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. To run unit tests: -1. Develop your unit tests. - * See HelloDslTest.groovy in the nf-hello plugin for unit test examples. +1. Develop your unit tests. See [HelloDslTest.groovy](https://github.com/nextflow-io/nf-hello/blob/gradle-plugin-example/src/test/groovy/nextflow/hello/HelloDslTest.groovy) in the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin for unit test examples. 2. In the plugin root directory, run `make test`. +(gradle-plugin-package)= -### Packaging, uploading, and publishing a plugin +## Packaging, uploading, and publishing a plugin The Gradle plugin for Nextflow plugins simplifies publishing your plugin. diff --git a/docs/index.md b/docs/index.md index 7943bd88b7..ae443012c5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -67,7 +67,6 @@ config executor cache-and-resume reports -plugins ``` ```{toctree} @@ -146,7 +145,6 @@ migrations/index developer/index developer/diagram developer/packages -developer/plugins ``` ```{toctree} diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index b424ff4233..c19cba3ef4 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -46,7 +46,7 @@ To migrate an existing Nextflow plugin: 3. Replace the contents of `settings.gradle` with the following: - ``` + ```groovy rootProject.name = '' ``` @@ -54,7 +54,7 @@ To migrate an existing Nextflow plugin: 4. In the project root, create a new `build.gradle` file with the following configuration: - ``` + ```groovy // Plugins plugins { id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' @@ -131,4 +131,4 @@ To migrate an existing Nextflow plugin: 6. Update `README.md` with information about the structure of your plugin. 7. In the plugin root directory, run `make assemble`. -The Gradle plugin for Nextflow plugins also supports publishing plugins. See Packaging, uploading, and publishing for more information. +The Gradle plugin for Nextflow plugins also supports publishing plugins. See {ref}`gradle-plugin-package` for more information. diff --git a/docs/plugins.md b/docs/plugins.md deleted file mode 100644 index 68d33151c1..0000000000 --- a/docs/plugins.md +++ /dev/null @@ -1,63 +0,0 @@ -(plugins-page)= - -# Plugins - -Nextflow has a plugin system that allows the use of extensible components that are downloaded and installed at runtime. - -(plugins-core)= - -## Core plugins - -The following functionalities are provided via plugin components, and they make part of the Nextflow *core* plugins: - -- `nf-amazon`: Support for Amazon Web Services. -- `nf-azure`: Support for Microsoft Azure. -- `nf-cloudcache`: Support for the cloud cache (see `NXF_CLOUDCACHE_PATH` under {ref}`config-env-vars`). -- `nf-console`: Implement Nextflow [REPL console](https://www.nextflow.io/blog/2015/introducing-nextflow-console.html). -- `nf-k8s`: Support for Kubernetes. -- `nf-google`: Support for Google Cloud. -- `nf-tower`: Support for [Seqera Platform](https://seqera.io) (formerly Tower Cloud). -- `nf-wave`: Support for [Wave containers](https://seqera.io/wave/) service. - -## Using plugins - -The core plugins do not require any configuration. They are automatically installed when the corresponding feature is requested by a Nextflow pipeline. You can still specify them as described below, e.g. if you want to pin the version of a plugin, however if you try to use a plugin version that isn't compatible with your Nextflow version, Nextflow will fail. - -You can enable a plugin by declaring it in your Nextflow configuration: - -```groovy -plugins { - id 'nf-hello@0.1.0' -} -``` - -Or you can use the `-plugins` command line option: - -```bash -nextflow run -plugins nf-hello@0.1.0 -``` - -The plugin identifier consists of the plugin name and plugin version separated by a `@`. Multiple plugins can be specified in the configuration with multiple `id` declarations, or on the command line as a comma-separated list. When specifying plugins via the command line, any plugin declarations in the configuration file are ignored. - -The plugin version is optional. If it is not specified, Nextflow will download the latest plugin version that is compatible with your Nextflow version. In general, it recommended that you not specify the plugin version unless you actually want to stick to that version, such as for [offline usage](#offline-usage). - -The core plugins are documented in this documentation. For all other plugins, please refer to the plugin's code repository for documentation and support. - -:::{versionadded} 25.02.0-edge -::: - -The plugin version can be prefixed with `~` to pin the major and minor version while allowing the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`, which is `2.9.2`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. - -## Offline usage - -To use Nextflow plugins in an offline environment: - -1. {ref}`Install Nextflow ` on a system with an internet connection. - -2. Download any additional plugins by running `nextflow plugin install `. Alternatively, simply run your pipeline once and Nextflow will download all of the plugins that it needs. - -3. Copy the `nextflow` binary and `$HOME/.nextflow` folder to your offline environment. - -4. In your Nextflow configuration file, specify each plugin that you downloaded, both name and version, including default plugins. This will prevent Nextflow from trying to download newer versions of plugins. - -Nextflow caches the plugins that it downloads, so as long as you keep using the same Nextflow version and pin your plugin versions in your config file, Nextflow will use the locally installed plugins and won't try to download them from the Internet. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 07b3c3fe2a..b56c841f10 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -1,4 +1,4 @@ -(developing-plugins-page)= +(dev-plugins-page)= # Developing plugins @@ -8,19 +8,19 @@ The Nextflow plugin framework streamlines plugin development by providing the st Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(developing-plugins-framework)= +(dev-plugins-framework)= ## Framework Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. -(developing-plugins-gradle)= +(dev-plugins-gradle)= ## Gradle plugin for Nextflow plugins -[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. +[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. -(developing-plugins-structure)= +(dev-plugins-structure)= ### Structure @@ -61,10 +61,10 @@ This structure contains the following key files and folders: See {ref}`nf-hello-page` for an example of a plugin built using this structure. :::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. See nf-hello for an example of a plugin with the old structure. +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(developing-plugins-make)= +(dev-plugins-make)= ### make commands @@ -83,18 +83,18 @@ make assemble The following `make` commands are available: `assemble` -: Compiles the Nextflow plugin code and assembles it into a zip file. See Creating a plugin for more information. +: Compiles the Nextflow plugin code and assembles it into a zip file. See {ref}`gradle-plugin-create` for more information. `test` -: Runs plugin unit tests. See Unit testing for more information. +: Runs plugin unit tests. See {ref}`gradle-plugin-unit-test` for more information. `install` -: Installs the plugin into the local nextflow plugins directory. See Running locally for more information. +: Installs the plugin into the local nextflow plugins directory. See {ref}`gradle-plugin-install` for more information. `release` -: Publishes the plugin. See Packaging, uploading, and publishing for more information. +: Publishes the plugin. See {ref}`gradle-plugin-package` for more information. -(developing-plugins-versioning)= +(dev-plugins-versioning)= ### Versioning @@ -108,7 +108,7 @@ plugins { See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. -(developing-plugins-extension)= +(dev-plugins-extension)= ## Extension points @@ -217,7 +217,7 @@ This approach is not required to support plugin config options. However, it allo Plugins can define custom executors that can be used with the `executor` process directive. -To implement an executor, create a class in your plugin that extends the [Executor](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: +To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy)class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: ```groovy import nextflow.executor.Executor @@ -249,9 +249,9 @@ See the source code of Nextflow's built-in executors for examples of how to impl ### Filesystems -Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see Remote files. +Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. -To implement a custom filesystem, create a class in your plugin that extends [FileSystemProvider](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: +To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example: ```groovy import java.nio.file.spi.FileSystemProvider @@ -371,9 +371,9 @@ The above snippet is based on the [nf-sqldb](https://github.com/nextflow-i ### Process directives -Plugins that implement a custom executor will likely need to access process directives that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. +Plugins that implement a custom executor will likely need to access {ref}`process directives ` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor. -Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the ext directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. +Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the {ref}`process-ext` directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines. For example, a custom executor can use existing process directives and a custom setting through the `ext` directive: @@ -451,6 +451,7 @@ myplugin.enabled = true See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. +(dev-plugins-env-var)= ## Environment variables diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 689b5f0a69..38e0b4d00e 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -`nf-hello` is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/master) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. -The nf-hello plugin has the following structure: +The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/master) has the following structure: ``` nf-hello diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 259d1e9e0c..5d229c08f7 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -28,7 +28,7 @@ Core plugins do not require configuration. The latest versions of core plugins a Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. :::{note} -The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See Environment variables for more information. +The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-var` for more information. :::

Third-party plugins

@@ -64,4 +64,4 @@ Version components: * PATCH: Increment for backward-compatible bug fixes. Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. -See Semantic Versioning for more information about the specification. +See {ref}`dev-plugins-versioning` for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 8aef78fc7e..da319a0849 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -1,7 +1,11 @@ +(using-plugins-page)= + # Using plugins Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin. +(using-plugins-identifiers)= + ## Identifiers A plugin identifier consists of the plugin name and version, separated by an `@` symbol: @@ -13,7 +17,7 @@ nf-hello@0.5.0 The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin. :::{note} -Plugin versions are required to run plugins in an Offline environment. +Plugin versions are required for {ref}`offline usage `. ::: :::{versionadded} 25.02.0-edge. @@ -25,6 +29,8 @@ The plugin version can be prefixed with `~` to pin the major and minor versions It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. ::: +(using-plugins-config)= + ## Configuration Plugins can be configured via nextflow configuration files or at runtime. @@ -57,13 +63,15 @@ Plugin declarations in nextflow configuration files are ignored when specifying When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. +(using-plugins-offline)= + ## Offline usage Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. To use Nextflow plugins in an offline environment: -1. Install a self-contained version of Nextflow in an environment with an internet connection. See Standalone distribution for more information. +1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information. 2. Run `nextflow plugin install @` to download the plugins. :::{tip} @@ -71,8 +79,8 @@ To use Nextflow plugins in an offline environment: ::: 3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. -4. Specify each plugin and its version in Nextflow configuration files or at runtime. See Configuration for more information. +4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information. :::{warning} - Nextflow will attempt to download newer versions of plugins if their versions are not set. See Plugin identifiers for more information. + Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information. ::: \ No newline at end of file diff --git a/docs/reference/env-vars.md b/docs/reference/env-vars.md index abce1e6548..bc507837ea 100644 --- a/docs/reference/env-vars.md +++ b/docs/reference/env-vars.md @@ -154,7 +154,7 @@ The following environment variables control the configuration of the Nextflow ru `NXF_PLUGINS_TEST_REPOSITORY` : :::{versionadded} 23.04.0 ::: -: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. See {ref}`testing-plugins` for more information. +: Defines a custom plugin registry or plugin release URL for testing plugins outside of the main registry. `NXF_PUBLISH_FAIL_ON_ERROR` : :::{versionadded} 24.04.3 diff --git a/docs/strict-syntax.md b/docs/strict-syntax.md index 19b8b71148..3f31201739 100644 --- a/docs/strict-syntax.md +++ b/docs/strict-syntax.md @@ -595,4 +595,4 @@ There are two ways to preserve Groovy code: Any Groovy code can be moved into the `lib` directory, which supports the full Groovy language. This approach is useful for temporarily preserving some Groovy code until it can be updated later and incorporated into a Nextflow script. See {ref}`lib-directory` documentation for more information. -For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`plugins-dev-page` for more information on how to develop plugins. +For Groovy code that is complicated or if it depends on third-party libraries, it may be better to create a plugin. Plugins can define custom functions that can be included by Nextflow scripts like a module. Furthermore, plugins can be easily re-used across different pipelines. See {ref}`dev-plugins-page` for more information on how to develop plugins. From 5471abcc7914cf504e8a3848e0a1b2bd93ac968f Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 29 Apr 2025 01:39:56 +1200 Subject: [PATCH 04/19] Add repo link Signed-off-by: Christopher Hakkaart --- docs/plugins/example-nf-hello.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index 38e0b4d00e..b727838163 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/master) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. -The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/master) has the following structure: +The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) has the following structure: ``` nf-hello @@ -56,3 +56,5 @@ The `nf-hello` plugin can be configured via nextflow configuration files or at r ```bash nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 ``` + +See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. From 3dfb9458b9365ffb4abf627046ad8c86693359fe Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 29 Apr 2025 09:38:39 +1200 Subject: [PATCH 05/19] Add Trace observers note Signed-off-by: Christopher Hakkaart --- docs/plugins/developing-plugins.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index b56c841f10..761b4182ea 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -398,6 +398,10 @@ class MyExecutor extends Executor { ### Trace observers +:::{versionchanged} 25.04 +The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead. +::: + A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers. Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example: From e254b63611384abf82ca5f28b2f242dd2bc2a175 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Mon, 5 May 2025 18:44:38 +1200 Subject: [PATCH 06/19] Revise defined registry Signed-off-by: Christopher Hakkaart --- docs/migrating-gradle-plugin.md | 2 -- docs/plugins/plugins.md | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index c19cba3ef4..384bc7b5d5 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -17,8 +17,6 @@ The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. -Developers can upload plugins to the plugin registry. The Gradle plugin for Nextflow plugins supports this process with Gradle tasks that package, upload, and publish plugins. - ## Impact on users and developers The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 5d229c08f7..59405d71ab 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -51,7 +51,7 @@ See {ref}`using-plugins-page` for more information. ## Nextflow plugin registry -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. Developers can upload plugins to the registry, with built-in support from the Gradle plugin for Nextflow plugins. +The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. ## Versioning From 5a6e4d9ac24bd78784f4c2be0907d88ad55c5651 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 6 May 2025 15:36:00 +1200 Subject: [PATCH 07/19] Add plugin create documentation Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 85 +++++----------------- docs/migrating-gradle-plugin.md | 20 ++--- docs/migrations/25-04.md | 6 ++ docs/plugins/developing-plugins.md | 113 ++++++++++++++++------------- docs/plugins/using-plugins.md | 2 +- docs/reference/cli.md | 6 +- 6 files changed, 104 insertions(+), 128 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 2eb45aa150..81f9303e88 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -2,7 +2,7 @@ # Using the Gradle plugin -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. This guide describes how to use the Gradle plugin for plugin development. +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development and incorporates the Gradle plugin by default. You can use the `nextflow plugin create` sub-command to create plugins scaffolds with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. :::{note} Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. @@ -12,71 +12,18 @@ Nextflow Plugins can be developed without the Gradle plugin. However, this appro ## Creating a plugin -The [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin uses the Gradle plugin and is a valuable starting point for developers. +:::{versionadded} 25.04.0 +::: To create a Nextflow plugin with the Gradle plugin: -1. Fork the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin. See {ref}`nf-hello-page` for more information. -2. Rename the forked `nf-hello` directory with your plugin name. -3. Replace the contents of `settings.gradle` with the following: - - ```groovy - rootProject.name = '' - ``` - - Replace `PLUGIN_NAME` with your plugin name. - -4. Replace the contents of `build.gradle` with the following: - - ```groovy - // Plugins - plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' - } - - // Dependencies (optional) - dependencies { - - } - - // Plugin version - version = '' - - nextflowPlugin { - // Minimum Nextflow version - nextflowVersion = '' - - // Plugin metadata - provider = '' - className = '' - extensionPoints = [ - '' - ] - - publishing { - github { - repository = '' - userName = project.findProperty('github_username') - authToken = project.findProperty('github_access_token') - email = project.findProperty('github_commit_email') - indexUrl = '' - } - } - } - ``` - - Replace the following: - - - `DEPENDENCY`: (optional) your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - - `PLUGIN_VERSION:` your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: the minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. - - `PROVIDER`: your name or organization—for example, `nextflow`. - - `CLASS_NAME`: your plugin class name—for example, `nextflow.hello.HelloPlugin`. - - `EXTENSION_POINT`: your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - - `GITHUB_REPOSITORY`: your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - - `GITHUB_INDEX_URL`: the URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). -5. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. -6. In the plugin root directory, run `make assemble`. +1. Run `nextflow plugin create`. + - When prompted `Enter plugin name:`, enter your plugin name. + - When prompted `Enter organization:`, enter your organization. + - When prompted `Enter project path:`, enter the project path in your local file system. + - When prompted `All good, are you OK to continue [y/N]?`, enter `y`. +2. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. +3. In the plugin root directory, run `make assemble`. (gradle-plugin-install)= @@ -92,13 +39,17 @@ To install a plugin locally: Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory. ::: -2. Configure your plugin. See {ref}`using-plugins-page` for more information. -3. Run your pipeline: +2. Run your pipeline: ```bash - nextflow run main.nf + nextflow run main.nf -plugins @ ``` + :::{note} + Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. + ::: + + (gradle-plugin-unit-test)= ## Unit testing a plugin @@ -107,7 +58,7 @@ Unit tests are small, focused tests designed to verify the behavior of individua To run unit tests: -1. Develop your unit tests. See [HelloDslTest.groovy](https://github.com/nextflow-io/nf-hello/blob/gradle-plugin-example/src/test/groovy/nextflow/hello/HelloDslTest.groovy) in the [nf-hello](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) plugin for unit test examples. +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template/tree/main) for unit test examples. 2. In the plugin root directory, run `make test`. (gradle-plugin-package)= diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 384bc7b5d5..3ca6889746 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -1,3 +1,5 @@ +(migrating-plugin-page)= + # Migrating to the Gradle plugin for Nextflow plugins This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. @@ -19,7 +21,7 @@ The Nextflow plugin registry is a centralized repository of assembled plugins. I ## Impact on users and developers -The impact of the Gradle plugin for Nextflow plugins and Nextflow plugin registry differs for plugin users and developers. +The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. ### Plugin Users @@ -36,7 +38,7 @@ To migrate an existing Nextflow plugin: - `nextflow.config` - `launch.sh` - `plugins/build.gradle` -2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. \ +2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. :::{note} Plugin sources should be in `src/main/groovy` or `src/main/java`. @@ -55,7 +57,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' } // Dependencies (optional) @@ -93,12 +95,12 @@ To migrate an existing Nextflow plugin: - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `24.11.0-edge`. - - `PROVIDER`: Your name or organization—for example, `nextflow`. - - `CLASS_NAME`: Your plugin class name—for example, `nextflow.hello.HelloPlugin`. - - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `nextflow.hello.HelloFactory`. - - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-hello`. - - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`plugins.json`](https://github.com/username/plugins/blob/main/plugins.json). + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.03.0-edge`. + - `PROVIDER`: Your name or organization—for example, `acme`. + - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. + - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. + - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-plugin-template`. + - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`https://github.com/nextflow-io/plugins/blob/main/plugins.json`](https://github.com/nextflow-io/plugins/blob/main/plugins.json). 5. Replace the contents of `Makefile` with the following: diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 205ed72e09..a1e64d75a9 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -37,6 +37,12 @@ The third preview of workflow outputs introduces the following breaking changes See {ref}`workflow-output-def` to learn more about the workflow output definition. +

Creating plugins

+ +The `nextflow plugin create` sub-command creates a scaffold for a Nextflow plugin using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). + +See {ref}`gradle-plugin-create` for details. + ## Enhancements

Improved inspect command

diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 761b4182ea..1663aed796 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -20,61 +20,15 @@ Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/p [Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. -(dev-plugins-structure)= - -### Structure - -Plugins built with the Gradle plugin for Nextflow plugins follow a standard project layout. This structure includes source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. - -A typical plugin built with the Gradle plugin for Nextflow plugins has the following structure: - -``` -. -├── COPYING -├── Makefile -├── README.md -├── build.gradle -├── gradle -│ └── wrapper -│ └── ... -├── gradlew -├── settings.gradle -└── src - ├── main - │ └── ... - └── test - └── ... -``` - -This structure contains the following key files and folders: - -- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. -- `src/main/`: The main source directory containing the plugin's implementation and resources.​ -- `src/test/`: The main source directory containing the plugin's unit testing. -- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ -- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ -- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ -- `build.gradle`: The primary build script for Gradle. -- `gradlew`: A script for executing the Gradle Wrapper. -- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. - -See {ref}`nf-hello-page` for an example of a plugin built using this structure. - -:::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: - -(dev-plugins-make)= - ### make commands -The Gradle plugin for Nextflow plugins defines tasks that can be executed with `./gradlew`. For example: +The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) defines tasks that can be executed with `./gradlew`. For example: ```bash ./gradlew assemble ``` -For convenience, the more important tasks are wrapped in a makefile and can be executed with the `make` command. For example: +For convenience, the most important tasks are wrapped in a `Makefile` and can be executed with the `make` command. For example: ```bash make assemble @@ -98,16 +52,75 @@ The following `make` commands are available: ### Versioning -The Gradle plugin is versioned and published to a [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' } ``` See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. +(dev-plugins-template)= + +## nf-plugin-template + +The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It incorporates the {ref}`dev-plugins-gradle` by default. You can use the `nextflow plugin create` sub-command to create plugins with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. See {ref}`gradle-plugin-create` for more information. + +(dev-plugins-structure)= + +### Structure + +Plugins built with [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) includes the [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle). The template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. + +Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) have the following structure: + +``` +. +├── build.gradle +├── COPYING +├── gradle +│ └── wrapper +│ └── ... +├── gradlew +├── Makefile +├── README.md +├── settings.gradle +└── src + ├── main + │ └── groovy + │ └── + │ └── plugin + │ └── ... + └── test + └── groovy + └── + └── plugin + └── ... +``` + +This structure contains the following key files and folders: + +- `build.gradle`: The primary build script for Gradle. +- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ +- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. +- `gradlew`: A script for executing the Gradle Wrapper. +- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ +- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ +- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. +- `src/main/groovy//plugin/`: The main source directory containing the plugin's implementation and resources.​ +- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit testing. + +See {ref}`nf-hello-page` for an example of a plugin built using this structure. + +:::{note} +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +::: + +(dev-plugins-make)= + + (dev-plugins-extension)= ## Extension points diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index da319a0849..282bd92b28 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -20,7 +20,7 @@ The plugin version is optional. If it is not specified, Nextflow will download t Plugin versions are required for {ref}`offline usage `. ::: -:::{versionadded} 25.02.0-edge. +:::{versionadded} 25.02.0-edge ::: The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 055ace3030..f92b2af868 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -910,7 +910,11 @@ Manage plugins and run plugin-specific commands. $ nextflow plugin [options] ``` -The `plugin` command provides several subcommands for managing and using plugins: +`create` + +: :::{versionadded} 25.04.0 + ::: +: Create a plugin scaffold using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. `install ` From 9053ca9b0b1e548b2c010e23ea3b299ff8848c9f Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 6 May 2025 12:41:09 -0500 Subject: [PATCH 08/19] Fix plugin config scope example Signed-off-by: Ben Sherman --- docs/plugins/developing-plugins.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 1663aed796..b43a268712 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -214,6 +214,10 @@ import nextflow.script.dsl.Description ''') class MyPluginConfig implements ConfigScope { + // no-arg constructor is required to enable validation of config options + MyPluginConfig() { + } + MyPluginConfig(Map opts) { this.createMessage = opts.createMessage } From ce87361006e0b76cdeac10aefd4189286506ca2b Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Wed, 7 May 2025 14:08:29 +1200 Subject: [PATCH 09/19] Only support registry Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 28 ++++++++++++---------------- docs/migrating-gradle-plugin.md | 21 ++++++++++----------- docs/plugins/developing-plugins.md | 2 +- docs/plugins/example-nf-hello.md | 2 +- 4 files changed, 24 insertions(+), 29 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 81f9303e88..d3412c432b 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -65,25 +65,21 @@ To run unit tests: ## Packaging, uploading, and publishing a plugin -The Gradle plugin for Nextflow plugins simplifies publishing your plugin. +The Gradle plugin for Nextflow plugins simplifies publishing your plugin to the Nextflow Plugin Registry. -To package, upload, and publish your plugin: +:::{note} +The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +::: -1. Fork the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins). -2. In the plugin root directory, open `build.gradle` and ensure that: - * `github.repository` matches the plugin repository. - * `github.indexUrl` matches your fork of the plugins index repository. -3. Create a file named `$HOME/.gradle/gradle.properties` and add the following: +To package, upload, and publish your plugin to the Nextflow Plugin Registry: + +1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. +2. Add the following properties: ```bash - github_username= - github_access_token= - github_commit_email= + pluginRegistry.accessToken= ``` - Replace the following: - * `GITHUB_USERNAME`: your GitHub username granting access to the plugin repository. - * `GITHUB_ACCESS_TOKEN`: your GitHub access token with permission to upload and commit changes to the plugin repository. - * `GITHUB_EMAIL`: your email address associated with your GitHub account. -4. Run `make release`. -5. Create a pull request against the [Nextfow plugins index repository](https://github.com/nextflow-io/plugins) from your fork. \ No newline at end of file + Replace with your plugin registry access token. + +3. Run `make release`. \ No newline at end of file diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 3ca6889746..75e59aaef4 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -15,10 +15,14 @@ The Gradle plugin for Nextflow plugins simplifies and standardizes the developme The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. -### Nextflow plugin registry +### Nextflow Plugin Registry The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. +:::{note} +The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +::: + ## Impact on users and developers The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. @@ -57,7 +61,7 @@ To migrate an existing Nextflow plugin: ```groovy // Plugins plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' } // Dependencies (optional) @@ -79,13 +83,10 @@ To migrate an existing Nextflow plugin: '' ] - publishing { - github { - repository = '' - userName = project.findProperty('github_username') - authToken = project.findProperty('github_access_token') - email = project.findProperty('github_commit_email') - indexUrl = '' + publishing { + registry { + url = 'https://nf-plugins-registry.dev-tower.net/api' + authToken = project.findProperty('pluginRegistry.accessToken') } } } @@ -99,8 +100,6 @@ To migrate an existing Nextflow plugin: - `PROVIDER`: Your name or organization—for example, `acme`. - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. - - `GITHUB_REPOSITORY`: Your GitHub plugin repository name—for example, `nextflow-io/nf-plugin-template`. - - `GITHUB_INDEX_URL`: The URL of your fork of the plugins index repository—for example, [`https://github.com/nextflow-io/plugins/blob/main/plugins.json`](https://github.com/nextflow-io/plugins/blob/main/plugins.json). 5. Replace the contents of `Makefile` with the following: diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 1663aed796..5a987dd6f5 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -56,7 +56,7 @@ The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https ```nextflow plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha3' + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' } ``` diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index b727838163..b643218eb8 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -54,7 +54,7 @@ It also includes several classes that demonstrate different plugin functionality The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: ```bash -nextflow run hello -plugins nf-hello@0.5.0,nf-amazon@2.9.0 +nextflow run hello -plugins nf-hello@0.5.0 ``` See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. From 2bb396034e28b70bbc3515372255b5131a2ae5f7 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 11:18:32 -0500 Subject: [PATCH 10/19] cleanup Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 47 +++++----- docs/index.md | 18 ++-- docs/migrating-gradle-plugin.md | 14 +-- docs/migrations/25-04.md | 2 +- docs/plugins/developing-plugins.md | 138 +++++++++++++---------------- docs/plugins/example-nf-hello.md | 25 ++---- docs/plugins/plugins.md | 33 +++---- docs/plugins/using-plugins.md | 27 +++--- docs/reference/stdlib-groovy.md | 2 +- 9 files changed, 140 insertions(+), 166 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index d3412c432b..4ee7e08366 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -2,10 +2,10 @@ # Using the Gradle plugin -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and incorporates custom Gradle tasks that streamline building, testing, and publishing Nextflow plugins. The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development and incorporates the Gradle plugin by default. You can use the `nextflow plugin create` sub-command to create plugins scaffolds with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. :::{note} -Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: (gradle-plugin-create)= @@ -15,21 +15,19 @@ Nextflow Plugins can be developed without the Gradle plugin. However, this appro :::{versionadded} 25.04.0 ::: -To create a Nextflow plugin with the Gradle plugin: +The easiest way to boostrap a Nextflow plugin based on the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). -1. Run `nextflow plugin create`. - - When prompted `Enter plugin name:`, enter your plugin name. - - When prompted `Enter organization:`, enter your organization. - - When prompted `Enter project path:`, enter the project path in your local file system. - - When prompted `All good, are you OK to continue [y/N]?`, enter `y`. -2. Develop your plugin extension points. See {ref}`dev-plugins-extension` for descriptions and examples. -3. In the plugin root directory, run `make assemble`. +To create a Nextflow plugin with the Gradle plugin, simply run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. + +See {ref}`dev-plugins-extension` for more information about developing a plugin. -(gradle-plugin-install)= +1. Run `nextflow plugin create`. +2. Develop your plugin extension points. +3. In the plugin root directory, run `make assemble`. -## Installing a plugin +## Building a plugin -Plugins can be installed locally without being packaged, uploaded, and published. +Plugins can be installed locally without being published. To install a plugin locally: @@ -49,37 +47,38 @@ To install a plugin locally: Plugins can also be configured via nextflow configuration files. See {ref}`using-plugins-page` for more information. ::: +(gradle-plugin-test)= -(gradle-plugin-unit-test)= - -## Unit testing a plugin +## Testing a plugin Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. To run unit tests: -1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template/tree/main) for unit test examples. +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template) for unit test examples. + 2. In the plugin root directory, run `make test`. -(gradle-plugin-package)= +(gradle-plugin-publish)= -## Packaging, uploading, and publishing a plugin +## Publishing a plugin -The Gradle plugin for Nextflow plugins simplifies publishing your plugin to the Nextflow Plugin Registry. +The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. :::{note} -The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. ::: -To package, upload, and publish your plugin to the Nextflow Plugin Registry: +To publish your plugin: 1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. + 2. Add the following properties: ```bash pluginRegistry.accessToken= ``` - Replace with your plugin registry access token. + Replace `` with your plugin registry access token. -3. Run `make release`. \ No newline at end of file +3. Run `make release`. diff --git a/docs/index.md b/docs/index.md index 635d6d6b3a..3d09716c36 100644 --- a/docs/index.md +++ b/docs/index.md @@ -146,23 +146,23 @@ migrations/index ```{toctree} :hidden: -:caption: Contributing +:caption: Plugins :maxdepth: 1 -developer/index -developer/diagram -developer/packages +plugins/plugins +plugins/using-plugins +plugins/developing-plugins +plugins/example-nf-hello ``` ```{toctree} :hidden: -:caption: Plugins +:caption: Contributing :maxdepth: 1 -plugins/plugins -plugins/using-plugins -plugins/developing-plugins -plugins/example-nf-hello +developer/index +developer/diagram +developer/packages ``` ```{toctree} diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 75e59aaef4..87187281f2 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -1,17 +1,17 @@ (migrating-plugin-page)= -# Migrating to the Gradle plugin for Nextflow plugins +# Migrating to the Nextflow Gradle plugin -This page introduces the Gradle plugin for Nextflow plugins, the Nextflow plugin registry, and how to migrate to the new plugin framework. +This page introduces the Nextflow Gradle plugin, the Nextflow plugin registry, and how to migrate to the new plugin framework. ## Improvements to the plugin framework The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. -### Gradle plugin for Nextflow plugins +### Nextflow Gradle plugin -The Gradle plugin for Nextflow plugins simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. +The Nextflow Gradle plugin simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. @@ -25,7 +25,7 @@ The Nextflow Plugin Registry is currently available as private beta technology. ## Impact on users and developers -The impact of the Gradle plugin for Nextflow plugins differs for plugin users and developers. +The impact of the Nextflow Gradle plugin differs for plugin users and developers. ### Plugin Users @@ -33,7 +33,7 @@ If you are a plugin user, no immediate actions are required. The plugin configur ### Plugin developers -Developers are encouraged to migrate to the Gradle plugin for Nextflow plugins and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. +Developers are encouraged to migrate to the Nextflow Gradle plugin and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. To migrate an existing Nextflow plugin: @@ -130,4 +130,4 @@ To migrate an existing Nextflow plugin: 6. Update `README.md` with information about the structure of your plugin. 7. In the plugin root directory, run `make assemble`. -The Gradle plugin for Nextflow plugins also supports publishing plugins. See {ref}`gradle-plugin-package` for more information. +The Nextflow Gradle plugin also supports publishing plugins. See {ref}`gradle-plugin-publish` for more information. diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 329ac5bc9d..0509d97a53 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -49,7 +49,7 @@ You can explore this lineage from the command line using the {ref}`cli-lineage` See the {ref}`cli-lineage` command and {ref}`config-lineage` config scope for details. -

Streamlined plugin development

+

Simplified plugin development

The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the new [plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 394b364919..470983506b 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -2,79 +2,23 @@ # Developing plugins -The Nextflow plugin framework streamlines plugin development by providing the structure and tools needed to extend Nextflow functionality. The Gradle plugin for Nextflow plugins simplifies development by configuring default Nextflow dependencies and incorporates custom Gradle tasks that streamline building and publishing plugins. - -:::{note} -Nextflow Plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. -::: - -(dev-plugins-framework)= - -## Framework - -Nextflow plugins use the [Plugin Framework for Java (p4fj)](https://github.com/pf4j/pf4j) framework to install, update, load, and unload plugins. p4fj creates a separate class loader for each plugin, allowing plugins to use their own versions of dependency jars. Nextflow defines several p4fj `ExtensionPoints` that plugin developers can extend. - -(dev-plugins-gradle)= - -## Gradle plugin for Nextflow plugins - -[Gradle](https://gradle.org/) is a flexible build automation tool optimized for Java and Groovy projects. The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) streamlines the development process by automatically setting up essential Nextflow dependencies and providing custom Gradle tasks to simplify plugin building and publishing. - -### make commands - -The [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle) defines tasks that can be executed with `./gradlew`. For example: - -```bash -./gradlew assemble -``` - -For convenience, the most important tasks are wrapped in a `Makefile` and can be executed with the `make` command. For example: - -```bash -make assemble -``` - -The following `make` commands are available: - -`assemble` -: Compiles the Nextflow plugin code and assembles it into a zip file. See {ref}`gradle-plugin-create` for more information. - -`test` -: Runs plugin unit tests. See {ref}`gradle-plugin-unit-test` for more information. - -`install` -: Installs the plugin into the local nextflow plugins directory. See {ref}`gradle-plugin-install` for more information. - -`release` -: Publishes the plugin. See {ref}`gradle-plugin-package` for more information. - -(dev-plugins-versioning)= - -### Versioning - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/). It can be declared and managed like any other dependency in the `build.gradle` file: - -```nextflow -plugins { - id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' -} -``` - -See the source code in [nextflow-plugin-gradle](https://github.com/nextflow-io/nextflow-plugin-gradle) for implementation details. +This page describes how to develop plugins for Nextflow. (dev-plugins-template)= -## nf-plugin-template +## Nextflow plugin template + +The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It uses [Gradle](https://gradle.org/), a build automation tool optimized for Java and Groovy projects, as well as the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle). -The [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It incorporates the {ref}`dev-plugins-gradle` by default. You can use the `nextflow plugin create` sub-command to create plugins with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) scaffold. See {ref}`gradle-plugin-create` for more information. +You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. (dev-plugins-structure)= ### Structure -Plugins built with [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) includes the [Gradle plugin for Nextflow plugins](https://github.com/nextflow-io/nextflow-plugin-gradle). The template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. +The plugin template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. -Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/) have the following structure: +Plugins built with the plugin template have the following structure: ``` . @@ -103,29 +47,70 @@ Plugins built with the [`nf-plugin-template`](https://github.com/nextflow-io/nf- This structure contains the following key files and folders: - `build.gradle`: The primary build script for Gradle. + - `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ + - `gradle/wrapper/`: Holds files related to the Gradle Wrapper. -- `gradlew`: A script for executing the Gradle Wrapper. + +- `gradlew`: The Gradle Wrapper script, which allows you to use Gradle without installing it into your environment. + - `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ + - `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ + - `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. + - `src/main/groovy//plugin/`: The main source directory containing the plugin's implementation and resources.​ -- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit testing. -See {ref}`nf-hello-page` for an example of a plugin built using this structure. +- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit tests. + +See {ref}`nf-hello-page` for an example of a plugin built with the plugin template. + +### Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by providing default configuration, Nextflow dependencies, and custom Gradle tasks for building and publishing plugins. + +It is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), and can be declared and managed like any other dependency in the `build.gradle` file: + +```nextflow +plugins { + id 'io.nextflow.nextflow-plugin' version '0.0.1-alpha4' +} +``` :::{note} Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. ::: -(dev-plugins-make)= +### Make commands + +The plugin template includes a Makefile which wraps the most important Gradle tasks provided by the Nextflow Gradle plugin. + +These tasks can be executed with [Make](https://www.gnu.org/software/make/). For example: + +```bash +make assemble +``` + +The following `make` commands are available: + +`assemble` +: Compiles the Nextflow plugin code and assembles it into a zip archive. + +`test` +: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. + +`install` +: Installs the plugin into the local Nextflow plugins directory. +`release` +: Publishes the plugin. See {ref}`gradle-plugin-publish` for more information. (dev-plugins-extension)= ## Extension points -Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how they can be used. +Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how to use them. ### Commands @@ -159,7 +144,7 @@ class MyPlugin extends BasePlugin implements PluginAbstractExec { The command can be run using the `nextflow plugin` command: -``` +```bash nextflow plugin my-plugin:hello --foo --bar ``` @@ -234,7 +219,7 @@ This approach is not required to support plugin config options. However, it allo Plugins can define custom executors that can be used with the `executor` process directive. -To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy)class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: +To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example: ```groovy import nextflow.executor.Executor @@ -263,7 +248,6 @@ process foo { See the source code of Nextflow's built-in executors for examples of how to implement various executor components. ::: - ### Filesystems Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`. @@ -290,7 +274,7 @@ You can then use this filesystem in your pipeline: input = file('myfs://') ``` -See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for examples of custom filesystems. +See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for an example of a custom filesystem. :::{tip} Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3. @@ -324,7 +308,7 @@ class MyExtension extends PluginExtensionPoint { You can then add this function to your pipeline: -``` +```nextflow include { reverseString } from 'plugin/my-plugin' channel.of( reverseString('hi') ) @@ -383,7 +367,11 @@ channel ``` :::{note} -The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`. +::: + +:::{tip} +Before creating a custom operator, consider whether the operator can be defined as a [function](#functions) that can be composed with existing operators such as `map` or `subscribe`. Functions are easier to implement and can be used anywhere in your pipeline, not just channel logic. ::: ### Process directives @@ -472,7 +460,7 @@ myplugin.enabled = true See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events. -(dev-plugins-env-var)= +(dev-plugins-env-vars)= ## Environment variables diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md index b643218eb8..99ca9984fc 100644 --- a/docs/plugins/example-nf-hello.md +++ b/docs/plugins/example-nf-hello.md @@ -2,9 +2,9 @@ # Example: nf-hello -[`nf-hello`](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) is a simple Nextflow plugin that uses the Gradle plugin for Nextflow plugins and is commonly used as a starting point for third-party plugin development. +[nf-hello](https://github.com/nextflow-io/nf-hello) is a simple Nextflow plugin that uses the Nextflow Gradle plugin and provides examples for various plugin extension points. -The [`nf-hello` plugin](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) has the following structure: +The `nf-hello` plugin has the following structure: ``` nf-hello @@ -36,20 +36,13 @@ nf-hello └── HelloFactoryTest.groovy ``` -It includes examples of different plugin extensions: +It includes the following extensions: -- A custom trace observer that prints a message when the workflow starts and when the workflow completes. -- A custom channel factory called reverse. -- A custom operator called goodbye. -- A custom function called randomString. - -It also includes several classes that demonstrate different plugin functionality: - -- `HelloConfig`: An example of how to handle options from the Nextflow configuration. -- `HelloExtension`: An example of how to create custom channel factories, operators, and functions that can be included in pipeline scripts. -- `HelloFactory`: An example of a workflow event with custom behavior. -- `HelloObserver`: An example of a workflow event with custom behavior. -- `HelloPlugin`: An example of a plugin entry point. +- A custom `hello` config scope (see `HelloConfig`). +- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`). +- A custom channel factory called `reverse` (see `HelloExtension`). +- A custom operator called `goodbye` (see `HelloExtension`). +- A custom function called `randomString` (see `HelloExtension`). The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: @@ -57,4 +50,4 @@ The `nf-hello` plugin can be configured via nextflow configuration files or at r nextflow run hello -plugins nf-hello@0.5.0 ``` -See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello/tree/gradle-plugin-example) for the plugin source code. +See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello) for the plugin source code. diff --git a/docs/plugins/plugins.md b/docs/plugins/plugins.md index 59405d71ab..470b8159a4 100644 --- a/docs/plugins/plugins.md +++ b/docs/plugins/plugins.md @@ -4,14 +4,9 @@ ## What are plugins -Nextflow plugins are extensions that enhance the functionality of the Nextflow workflow framework. They allow users to add new capabilities and integrate with external services without modifying core Nextflow code. +Nextflow plugins are extensions that enhance the functionality of Nextflow. They allow users to add new capabilities and integrate with external services without bloating their pipeline code or modifying Nextflow itself. -There are two types of Nextflow plugins: - -* Core plugins -* Third-party plugins - -The main features of each plugin type are described below. +There are two types of Nextflow plugins: core plugins and third-party plugins. The main features of each plugin type are described below.

Core plugins

@@ -20,7 +15,7 @@ Core plugins do not require configuration. The latest versions of core plugins a * `nf-amazon`: Support for Amazon Web Services. * `nf-azure`: Support for Microsoft Azure. * `nf-cloudcache`: Support for the cloud cache. -* `nf-console`: Implement Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). +* `nf-console`: Implementation of the Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/). * `nf-google`: Support for Google Cloud. * `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/). * `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/). @@ -28,14 +23,14 @@ Core plugins do not require configuration. The latest versions of core plugins a Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information. :::{note} -The automatic application of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-var` for more information. +The automatic retrieval of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-vars` for more information. :::

Third-party plugins

Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: -``` +```groovy plugins { id 'nf-hello@0.5.0' } @@ -43,7 +38,7 @@ plugins { To configure plugins at runtime, use the `-plugins` option. For example: -``` +```bash nextflow run main.nf -plugins nf-hello@0.5.0 ``` @@ -51,17 +46,23 @@ See {ref}`using-plugins-page` for more information. ## Nextflow plugin registry -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow is able to automatically locate and download configured plugins. +:::{versionadded} 25.04.0 +::: + +:::{note} +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. +::: + +The Nextflow plugin registry is a central repository for publishing and discovering Nextflow plugins. Nextflow is able to use the plugin registry to automatically find and download plugins at runtime. ## Versioning -Nextflow plugins follow the Semantic Versioning format: MAJOR.MINOR.PATCH (e.g., 0.5.0). This helps developers communicate the nature of changes in a release. +Nextflow plugins are free to use any versioning convention. Many plugins, including all core plugins, use [Semantic Versioning](https://semver.org/), which helps developers communicate the kind of changes in a release. -Version components: +Semantic versions have the form MAJOR.MINOR.PATCH (e.g., 0.5.0): -* MAJOR: Increment for incompatible changes. +* MAJOR: Increment for backward-incompatible changes. * MINOR: Increment for backward-compatible feature additions. * PATCH: Increment for backward-compatible bug fixes. Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format. -See {ref}`dev-plugins-versioning` for more information about the specification. diff --git a/docs/plugins/using-plugins.md b/docs/plugins/using-plugins.md index 282bd92b28..4e21292cc7 100644 --- a/docs/plugins/using-plugins.md +++ b/docs/plugins/using-plugins.md @@ -10,7 +10,7 @@ Nextflow core plugins require no additional configuration. When a pipeline uses A plugin identifier consists of the plugin name and version, separated by an `@` symbol: -```console +``` nf-hello@0.5.0 ``` @@ -26,61 +26,54 @@ Plugin versions are required for {ref}`offline usage `. The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`. :::{tip} -It is recommended to pin plugin versions to the major and minor versions and allow the latest patch update. +It is recommended to pin the major and minor version of each plugin that you use, in order to minimize the risk of breaking changes while allowing patch updates to be used automatically. ::: (using-plugins-config)= ## Configuration -Plugins can be configured via nextflow configuration files or at runtime. +Plugins can be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example: ```nextflow plugins { id 'nf-hello@0.5.0' + id 'nf-amazon@2.9.0' } ``` To configure plugins at runtime, use the `-plugins` option. For example: -```bash -nextflow run main.nf -plugins nf-hello@0.5.0 -``` - -To configure multiple plugins at runtime, use the `-plugins` option and a comma-separated list. For example: - ```bash nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0 ``` :::{note} -Plugin declarations in nextflow configuration files are ignored when specifying plugins via the `-plugins` option. +Plugin declarations in Nextflow configuration files are ignored when specifying plugins via the `-plugins` option. ::: ## Caching -When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). It tracks the plugins and their versions in a local cache located at `.nextflow/cache/`. If the Nextflow version and plugin versions match those from a previous run, the cached plugins are reused. +When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default). (using-plugins-offline)= ## Offline usage -Nextflow plugins may be required by some pipelines in an offline environment. Plugins must be manually downloaded and moved to the offline environment. +When running Nextflow in an offline environment, any required plugins must be downloaded and moved into the offline environment prior to any runs. To use Nextflow plugins in an offline environment: 1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information. -2. Run `nextflow plugin install @` to download the plugins. - :::{tip} - Running a pipeline in an environment with an internet connection will also download the plugins used by that pipeline. - ::: +2. Run `nextflow plugin install @` for each required plugin to download it. Alternatively, run the pipeline once, which will automatically download all plugins required by the pipeline. 3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment. + 4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information. :::{warning} Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information. - ::: \ No newline at end of file + ::: diff --git a/docs/reference/stdlib-groovy.md b/docs/reference/stdlib-groovy.md index 2d0bb83c8a..add202a445 100644 --- a/docs/reference/stdlib-groovy.md +++ b/docs/reference/stdlib-groovy.md @@ -21,5 +21,5 @@ println groovy.json.JsonOutput.toJson(vals) ``` :::{note} -The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. +The set of classes in Nextflow's runtime classpath can change between different Nextflow versions. As a best practice, any code that uses classes outside the Nextflow standard library should either be refactored to only use the Nextflow standard library or be refactored as a {ref}`plugin ` with explicit dependencies. ::: From e153d962808f8a5fb5fb91f99d3b91fdb1820007 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 11:34:22 -0500 Subject: [PATCH 11/19] Remove registry URL Signed-off-by: Ben Sherman --- docs/migrating-gradle-plugin.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 87187281f2..3280d36b62 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -83,9 +83,8 @@ To migrate an existing Nextflow plugin: '' ] - publishing { + publishing { registry { - url = 'https://nf-plugins-registry.dev-tower.net/api' authToken = project.findProperty('pluginRegistry.accessToken') } } From 1be033668dd4f4b9f3557b9b36f1f0b2aad187da Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 11:43:49 -0500 Subject: [PATCH 12/19] cleanup migration guide Signed-off-by: Ben Sherman --- docs/migrating-gradle-plugin.md | 38 +++++++++++++----------------- docs/plugins/developing-plugins.md | 2 +- docs/reference/cli.md | 2 +- 3 files changed, 18 insertions(+), 24 deletions(-) diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-gradle-plugin.md index 3280d36b62..ab07a0ce01 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-gradle-plugin.md @@ -1,47 +1,40 @@ (migrating-plugin-page)= -# Migrating to the Nextflow Gradle plugin +# Migrating to the Nextflow plugin registry -This page introduces the Nextflow Gradle plugin, the Nextflow plugin registry, and how to migrate to the new plugin framework. +The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. +## Overview -## Improvements to the plugin framework +### Nextflow plugin registry -The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying plugin development, streamlining publishing and discovery, and improving how plugins are loaded into workflows. These improvements make plugins more accessible, maintainable, and interoperable with Nextflow. - -### Nextflow Gradle plugin - -The Nextflow Gradle plugin simplifies and standardizes the development of Nextflow plugins. It configures default dependencies required for Nextflow integration and introduces custom Gradle tasks to streamline building, testing, packaging, and publishing plugins. - -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. - -### Nextflow Plugin Registry - -The Nextflow plugin registry is a centralized repository of assembled plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. Nextflow will automatically locate and download configured plugins. +The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. It is intended as a replacement for the [plugins index](https://github.com/nextflow-io/plugins) hosted on GitHub. :::{note} -The Nextflow Plugin Registry is currently available as private beta technology. Contact [info@nextflow.io](mailto:info@nextflow.io) to learn how to get access. +The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. ::: -## Impact on users and developers +### Nextflow Gradle plugin + +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. -The impact of the Nextflow Gradle plugin differs for plugin users and developers. +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. -### Plugin Users +## Impact on plugin users If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. -### Plugin developers +## Impact on plugin developers -Developers are encouraged to migrate to the Nextflow Gradle plugin and benefit from features that simplify plugin development and integration with the wider plugin ecosystem. +Developers are encouraged to migrate to the Nextflow Gradle plugin in order to take advantage of the simplified development and publishing process. To migrate an existing Nextflow plugin: 1. Remove the following files and folders: - `buildSrc/` - - `nextflow.config` - `launch.sh` - `plugins/build.gradle` + 2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. :::{note} @@ -95,7 +88,7 @@ To migrate an existing Nextflow plugin: - `DEPENDENCY`: (Optional) Your plugins dependency libraries—for example, `commons-io:commons-io:2.18.0`. - `PLUGIN_VERSION:` Your plugin version—for example, `0.5.0`. - - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.03.0-edge`. + - `MINIMUM_NEXTFLOW_VERSION`: The minimum Nextflow version required to run your plugin—for example, `25.04.0`. - `PROVIDER`: Your name or organization—for example, `acme`. - `CLASS_NAME`: Your plugin class name—for example, `acme.plugin.MyPlugin`. - `EXTENSION_POINT`: Your extension point identifiers that the plugin will implement or expose—for example, `acme.plugin.MyFactory`. @@ -127,6 +120,7 @@ To migrate an existing Nextflow plugin: ``` 6. Update `README.md` with information about the structure of your plugin. + 7. In the plugin root directory, run `make assemble`. The Nextflow Gradle plugin also supports publishing plugins. See {ref}`gradle-plugin-publish` for more information. diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 470983506b..ddce06d875 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -68,7 +68,7 @@ See {ref}`nf-hello-page` for an example of a plugin built with the plugin templa ### Nextflow Gradle plugin -The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by providing default configuration, Nextflow dependencies, and custom Gradle tasks for building and publishing plugins. +The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. It is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), and can be declared and managed like any other dependency in the `build.gradle` file: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 758f9b20d2..911bb0390b 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -996,7 +996,7 @@ $ nextflow plugin [options] : :::{versionadded} 25.04.0 ::: -: Create a plugin scaffold using [`nf-plugin-template`](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. +: Create a plugin scaffold using the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`gradle-plugin-create` for more information. `install ` From 7bfc78da9b4659cdd069ba6567616c1b05552016 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Wed, 7 May 2025 12:06:01 -0500 Subject: [PATCH 13/19] cleanup gradle plugin guide Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 22 ++++++++++++---------- docs/plugins/developing-plugins.md | 6 +++--- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 4ee7e08366..90701d7f8d 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -1,6 +1,6 @@ (gradle-plugin-page)= -# Using the Gradle plugin +# Using the Nextflow Gradle plugin The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. @@ -15,21 +15,17 @@ Nextflow plugins can be developed without the Gradle plugin. However, this appro :::{versionadded} 25.04.0 ::: -The easiest way to boostrap a Nextflow plugin based on the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). +The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. To create a Nextflow plugin with the Gradle plugin, simply run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. -See {ref}`dev-plugins-extension` for more information about developing a plugin. - -1. Run `nextflow plugin create`. -2. Develop your plugin extension points. -3. In the plugin root directory, run `make assemble`. +See {ref}`dev-plugins-extension` for more information about available plugin extension points. ## Building a plugin -Plugins can be installed locally without being published. +To build a plugin, simply run `make assemble`. -To install a plugin locally: +Plugins can also be installed locally without being published. To install a plugin locally: 1. In the plugin root directory, run `make install`. @@ -51,7 +47,9 @@ To install a plugin locally: ## Testing a plugin -Unit tests are small, focused tests designed to verify the behavior of individual plugin components and are an important part of software development. +### Unit tests + +Unit tests are small, focused tests designed to verify the behavior of individual plugin components. To run unit tests: @@ -59,6 +57,10 @@ To run unit tests: 2. In the plugin root directory, run `make test`. +### End-to-end tests + +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. See [nf-hello](https://github.com/nextflow-io/nf-hello) for an example of an end-to-end test for a Nextflow plugin. + (gradle-plugin-publish)= ## Publishing a plugin diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index ddce06d875..ecaa9740be 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -97,15 +97,15 @@ The following `make` commands are available: `assemble` : Compiles the Nextflow plugin code and assembles it into a zip archive. -`test` -: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. - `install` : Installs the plugin into the local Nextflow plugins directory. `release` : Publishes the plugin. See {ref}`gradle-plugin-publish` for more information. +`test` +: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. + (dev-plugins-extension)= ## Extension points From fa2dc0fb2e4da8c9aa4aec31ef7b90a52b0ae4c0 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Thu, 8 May 2025 10:09:45 +1200 Subject: [PATCH 14/19] Unlist h3 headings Signed-off-by: Christopher Hakkaart --- docs/gradle-plugin.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index 90701d7f8d..eaef8af07e 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -47,7 +47,7 @@ Plugins can also be installed locally without being published. To install a plug ## Testing a plugin -### Unit tests +

Unit tests

Unit tests are small, focused tests designed to verify the behavior of individual plugin components. @@ -57,7 +57,7 @@ To run unit tests: 2. In the plugin root directory, run `make test`. -### End-to-end tests +

End-to-end tests

End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. See [nf-hello](https://github.com/nextflow-io/nf-hello) for an example of an end-to-end test for a Nextflow plugin. From aa87a220520ec8c574f188523514abd5c3ff3003 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 9 May 2025 15:37:58 -0500 Subject: [PATCH 15/19] Replace nf-hello example with plugin template Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 6 +-- docs/index.md | 1 - docs/plugins/developing-plugins.md | 69 ++++++++++++++++++------------ docs/plugins/example-nf-hello.md | 53 ----------------------- 4 files changed, 45 insertions(+), 84 deletions(-) delete mode 100644 docs/plugins/example-nf-hello.md diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index eaef8af07e..cc479a095d 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -53,13 +53,13 @@ Unit tests are small, focused tests designed to verify the behavior of individua To run unit tests: -1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template) for unit test examples. +1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example unit test. 2. In the plugin root directory, run `make test`.

End-to-end tests

-End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. See [nf-hello](https://github.com/nextflow-io/nf-hello) for an example of an end-to-end test for a Nextflow plugin. +End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. (gradle-plugin-publish)= @@ -77,7 +77,7 @@ To publish your plugin: 2. Add the following properties: - ```bash + ``` pluginRegistry.accessToken= ``` diff --git a/docs/index.md b/docs/index.md index a5a690d1ad..e5c35652f1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -152,7 +152,6 @@ migrations/index plugins/plugins plugins/using-plugins plugins/developing-plugins -plugins/example-nf-hello ``` ```{toctree} diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index ecaa9740be..5fb21b689a 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -18,53 +18,68 @@ You can use the `nextflow plugin create` sub-command to create plugins from the The plugin template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy. -Plugins built with the plugin template have the following structure: +For example, a plugin created from the plugin template with the name `nf-hello` and organization `nextflow` will have the following structure: -``` -. -├── build.gradle +```console +nf-hello ├── COPYING +├── Makefile +├── README.md +├── build.gradle ├── gradle │ └── wrapper -│ └── ... +│ ├── gradle-wrapper.jar +│ └── gradle-wrapper.properties ├── gradlew -├── Makefile -├── README.md ├── settings.gradle -└── src - ├── main - │ └── groovy - │ └── - │ └── plugin - │ └── ... - └── test - └── groovy - └── - └── plugin - └── ... +├── src +│ ├── main +│ │ └── groovy +│ │ └── nextflow +│ │ └── hello +│ │ ├── HelloExtension.groovy +│ │ ├── HelloFactory.groovy +│ │ ├── HelloObserver.groovy +│ │ └── HelloPlugin.groovy +│ └── test +│ └── groovy +│ └── nextflow +│ └── hello +│ └── HelloObserverTest.groovy +└── validation + └── main.nf + └── nextflow.config ``` This structure contains the following key files and folders: -- `build.gradle`: The primary build script for Gradle. +- `.github/workflows`: GitHub Action which implements continuous integration for the plugin. + +- `build.gradle`: The Gradle build script. -- `COPYING`: Contains the licensing information for the project, detailing the terms under which the code can be used and distributed.​ +- `COPYING`: The project license, detailing the terms under which the code can be used and distributed.​ -- `gradle/wrapper/`: Holds files related to the Gradle Wrapper. +- `gradle/wrapper/`: Helper files for the Gradle Wrapper. - `gradlew`: The Gradle Wrapper script, which allows you to use Gradle without installing it into your environment. -- `Makefile`: Defines a set of tasks and commands for building and managing the project with the `make` automation tool.​ +- `Makefile`: Defines common tasks for building, testing, and publishing the plugin with Make.​ + +- `README.md`: The project README, which provides an overview of the project, including its purpose, features, and instructions for usage and development.​ + +- `settings.gradle`: The Gradle project configuration, which specifies project-specific settings such as the project name and included modules. + +- `src/main/groovy///`: The main source directory, which contains the plugin source code and resources.​ -- `README.md`: Provides an overview of the project, including its purpose, features, and instructions for usage and development.​ +- `src/test/groovy///`: The test source directory, which contains the plugin unit tests. -- `settings.gradle`: Configures the Gradle build, specifying project-specific settings such as the project name and included modules. +- `validation`: A small Nextflow pipeline which serves as an end-to-end test for the plugin. -- `src/main/groovy//plugin/`: The main source directory containing the plugin's implementation and resources.​ +The plugin template also implements the following example features: -- `src/test/groovy//plugin/`: The main source directory containing the plugin's unit tests. +- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`). -See {ref}`nf-hello-page` for an example of a plugin built with the plugin template. +- A custom function called `sayHello` (see `HelloExtension`). ### Nextflow Gradle plugin diff --git a/docs/plugins/example-nf-hello.md b/docs/plugins/example-nf-hello.md deleted file mode 100644 index 99ca9984fc..0000000000 --- a/docs/plugins/example-nf-hello.md +++ /dev/null @@ -1,53 +0,0 @@ -(nf-hello-page)= - -# Example: nf-hello - -[nf-hello](https://github.com/nextflow-io/nf-hello) is a simple Nextflow plugin that uses the Nextflow Gradle plugin and provides examples for various plugin extension points. - -The `nf-hello` plugin has the following structure: - -``` -nf-hello -├── COPYING -├── Makefile -├── README.md -├── build.gradle -├── gradle -│ └── wrapper -│ ├── gradle-wrapper.jar -│ └── gradle-wrapper.properties -├── gradlew -├── settings.gradle -└── src - ├── main - │ └── groovy - │ └── nextflow - │ └── hello - │ ├── HelloConfig.groovy - │ ├── HelloExtension.groovy - │ ├── HelloFactory.groovy - │ ├── HelloObserver.groovy - │ └── HelloPlugin.groovy - └── test - └── groovy - └── nextflow - └── hello - ├── HelloDslTest.groovy - └── HelloFactoryTest.groovy -``` - -It includes the following extensions: - -- A custom `hello` config scope (see `HelloConfig`). -- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`). -- A custom channel factory called `reverse` (see `HelloExtension`). -- A custom operator called `goodbye` (see `HelloExtension`). -- A custom function called `randomString` (see `HelloExtension`). - -The `nf-hello` plugin can be configured via nextflow configuration files or at runtime. For example: - -```bash -nextflow run hello -plugins nf-hello@0.5.0 -``` - -See the [nf-hello plugin repository](https://github.com/nextflow-io/nf-hello) for the plugin source code. From 758b8186c1efa9266feabe50c9bc3ffb9bca72ed Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 9 May 2025 16:09:42 -0500 Subject: [PATCH 16/19] Cleanup private beta notes Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 8 ++------ docs/index.md | 2 +- ...adle-plugin.md => migrating-plugin-registry.md} | 14 +++++++------- docs/migrations/25-04.md | 8 ++++++-- docs/plugins/developing-plugins.md | 6 ++++-- docs/process.md | 2 +- 6 files changed, 21 insertions(+), 19 deletions(-) rename docs/{migrating-gradle-plugin.md => migrating-plugin-registry.md} (88%) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index cc479a095d..cd0069a363 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -5,7 +5,7 @@ The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies plugin development by configuring default dependencies needed for Nextflow integration and defining Gradle tasks for building, testing, and publishing Nextflow plugins. :::{note} -Nextflow plugins can be developed without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin. +The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. ::: (gradle-plugin-create)= @@ -19,7 +19,7 @@ The easiest way to get started with the Nextflow Gradle plugin is to use the `ne To create a Nextflow plugin with the Gradle plugin, simply run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. -See {ref}`dev-plugins-extension` for more information about available plugin extension points. +See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. See {ref}`dev-plugins-extension-points` for more information about using plugin extension points. ## Building a plugin @@ -67,10 +67,6 @@ End-to-end tests are comprehensive tests that verify the behavior of an entire p The Nextflow Gradle plugin allows you to publish your plugin to the Nextflow plugin registry from the command line. -:::{note} -The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. -::: - To publish your plugin: 1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. diff --git a/docs/index.md b/docs/index.md index e5c35652f1..fe9a8101b6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -171,7 +171,7 @@ developer/packages data-lineage gradle-plugin -migrating-gradle-plugin +migrating-plugin-registry updating-spot-retries metrics flux diff --git a/docs/migrating-gradle-plugin.md b/docs/migrating-plugin-registry.md similarity index 88% rename from docs/migrating-gradle-plugin.md rename to docs/migrating-plugin-registry.md index ab07a0ce01..de1bf57fd9 100644 --- a/docs/migrating-gradle-plugin.md +++ b/docs/migrating-plugin-registry.md @@ -4,15 +4,15 @@ The Nextflow plugin ecosystem is evolving to support a more robust and user-friendly experience by simplifying the development, publishing, and discovery of Nextflow plugins. This page introduces the Nextflow plugin registry, the Nextflow Gradle plugin, and how to migrate to them. +:::{note} +The Nextflow plugin registry and Gradle plugin are currently available as a private beta. Plugin developers are encouraged to contact [info@nextflow.io](mailto:info@nextflow.io) for more information about accessing the registry. +::: + ## Overview ### Nextflow plugin registry -The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. The registry is integrated with the Nextflow runtime. It is intended as a replacement for the [plugins index](https://github.com/nextflow-io/plugins) hosted on GitHub. - -:::{note} -The Nextflow plugin registry is currently available as a private beta. Contact [info@nextflow.io](mailto:info@nextflow.io) for more information. -::: +The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. Nextflow 25.04 and later can use the plugin registry instead of the [legacy plugins index](https://github.com/nextflow-io/plugins) hosted on GitHub. ### Nextflow Gradle plugin @@ -35,7 +35,7 @@ To migrate an existing Nextflow plugin: - `launch.sh` - `plugins/build.gradle` -2. If your plugin uses a `plugins` directory, move the `src` directory to the project root. +2. If your plugin has a `plugins` directory, move the `src` directory to the project root. :::{note} Plugin sources should be in `src/main/groovy` or `src/main/java`. @@ -95,7 +95,7 @@ To migrate an existing Nextflow plugin: 5. Replace the contents of `Makefile` with the following: - ``` + ```Makefile # Build the plugin assemble: ./gradlew assemble diff --git a/docs/migrations/25-04.md b/docs/migrations/25-04.md index 383669f285..8080b6c156 100644 --- a/docs/migrations/25-04.md +++ b/docs/migrations/25-04.md @@ -52,9 +52,13 @@ See the {ref}`data-lineage-page` guide to get started.

Simplified plugin development

-The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the new [plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. +The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which is simpler and easier to maintain than the previous plugin boilerplate. -See {ref}`gradle-plugin-create` for details. +See the {ref}`gradle-plugin-page` guide for details. + +:::{note} +The Nextflow Gradle plugin and plugin registry are currently available as a private beta. See the {ref}`migration guide ` for more information. +::: ## Enhancements diff --git a/docs/plugins/developing-plugins.md b/docs/plugins/developing-plugins.md index 5fb21b689a..c48417660b 100644 --- a/docs/plugins/developing-plugins.md +++ b/docs/plugins/developing-plugins.md @@ -12,7 +12,9 @@ The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template You can use the `nextflow plugin create` sub-command to create plugins from the plugin template. See {ref}`gradle-plugin-create` for more information. -(dev-plugins-structure)= +:::{note} +The Nextflow Gradle plugin is currently available as a private beta. See the {ref}`migration guide ` for more information. +::: ### Structure @@ -121,7 +123,7 @@ The following `make` commands are available: `test` : Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information. -(dev-plugins-extension)= +(dev-plugins-extension-points)= ## Extension points diff --git a/docs/process.md b/docs/process.md index 78e9b35784..4eeb060a96 100644 --- a/docs/process.md +++ b/docs/process.md @@ -1164,7 +1164,7 @@ While this option can be used with any process output, it cannot be applied to i ## When -:::{node} +:::{note} As a best practice, conditional logic should be implemented in the calling workflow (e.g. using an `if` statement or {ref}`operator-filter` operator) instead of the process definition. ::: From e0866bce3e75d5a44eeec00b5313e4d1838ba5c0 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 9 May 2025 16:56:56 -0500 Subject: [PATCH 17/19] Add transition plan for legacy index -> registry Signed-off-by: Ben Sherman --- docs/migrating-plugin-registry.md | 42 ++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 4 deletions(-) diff --git a/docs/migrating-plugin-registry.md b/docs/migrating-plugin-registry.md index de1bf57fd9..3fc42c49ee 100644 --- a/docs/migrating-plugin-registry.md +++ b/docs/migrating-plugin-registry.md @@ -12,7 +12,7 @@ The Nextflow plugin registry and Gradle plugin are currently available as a priv ### Nextflow plugin registry -The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. Nextflow 25.04 and later can use the plugin registry instead of the [legacy plugins index](https://github.com/nextflow-io/plugins) hosted on GitHub. +The Nextflow plugin registry is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking. Nextflow 25.04 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. ### Nextflow Gradle plugin @@ -20,13 +20,41 @@ The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-grad The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. +## Timeline + +The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprecated in favor of the Nextflow plugin registry. + +:::{note} +The following timeline is tentative and subject to modification. +::: + +### Nextflow 25.04 + +The Nextflow plugin registry is available as a private beta. Nextflow 25.04 can use the Nextflow plugin registry as an opt-in feature. The Nextflow plugin registry will be automatically kept up-to-date with the [legacy plugin index](https://github.com/nextflow-io/plugins). + +During this time, plugin developers are encouraged to experiment with the Gradle plugin and plugin registry. + +### Nextflow 25.10 + +The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. + +Developers will be required to publish to the Nextflow plugin registry instead. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. + +### Nextflow 26.04 + +Nextflow 25.10 will only be able to use the Nextflow plugin registry. + +At some point in the future, the legacy plugin index will be **frozen** -- it will no longer receives updates from the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will remain available indefinitely. + ## Impact on plugin users -If you are a plugin user, no immediate actions are required. The plugin configuration has not changed. +No immediate actions are required for plugin users. The plugin configuration has not changed. ## Impact on plugin developers -Developers are encouraged to migrate to the Nextflow Gradle plugin in order to take advantage of the simplified development and publishing process. +Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The easiest way to do this is to migrate to the Nextflow Gradle plugin, which simplifies the development process and supports publishing to the plugin registry from the command line. + +### Migrating to the Nextflow Gradle plugin To migrate an existing Nextflow plugin: @@ -123,4 +151,10 @@ To migrate an existing Nextflow plugin: 7. In the plugin root directory, run `make assemble`. -The Nextflow Gradle plugin also supports publishing plugins. See {ref}`gradle-plugin-publish` for more information. +Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template, add your existing plugin code as needed. See {ref}`dev-plugins-template` for more information about the plugin template. + +### Publishing to the Nextflow plugin registry + +The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information. + +Once you migrate to the Gradle plugin, you will no longer be able to publish to the legacy plugin index. See the [transition timeline](#timeline) for more information. From de7aaa12f6d0098eff9960fc9b4337140b8bf324 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 9 May 2025 21:25:37 -0500 Subject: [PATCH 18/19] Apply suggestions from code review Signed-off-by: Ben Sherman --- docs/gradle-plugin.md | 8 ++++---- docs/migrating-plugin-registry.md | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/gradle-plugin.md b/docs/gradle-plugin.md index cd0069a363..2f9277e898 100644 --- a/docs/gradle-plugin.md +++ b/docs/gradle-plugin.md @@ -17,13 +17,13 @@ The Nextflow Gradle plugin and plugin registry are currently available as a priv The easiest way to get started with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command, which creates a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/), which in turn uses the Gradle plugin. -To create a Nextflow plugin with the Gradle plugin, simply run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. +To create a Nextflow plugin with the Gradle plugin, run `nextflow plugin create` on the command line. It will prompt you for your plugin name, organization name, and project path. See {ref}`dev-plugins-template` for more information about the Nextflow plugin template. See {ref}`dev-plugins-extension-points` for more information about using plugin extension points. ## Building a plugin -To build a plugin, simply run `make assemble`. +To build a plugin, run `make assemble`. Plugins can also be installed locally without being published. To install a plugin locally: @@ -47,7 +47,7 @@ Plugins can also be installed locally without being published. To install a plug ## Testing a plugin -

Unit tests

+

Unit tests

Unit tests are small, focused tests designed to verify the behavior of individual plugin components. @@ -57,7 +57,7 @@ To run unit tests: 2. In the plugin root directory, run `make test`. -

End-to-end tests

+

End-to-end tests

End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in a Nextflow pipeline. End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test. diff --git a/docs/migrating-plugin-registry.md b/docs/migrating-plugin-registry.md index 3fc42c49ee..eacff3a0ed 100644 --- a/docs/migrating-plugin-registry.md +++ b/docs/migrating-plugin-registry.md @@ -18,7 +18,7 @@ The Nextflow plugin registry is a central repository for Nextflow plugins. It ho The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins. -The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, this Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. +The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of ongoing improvements to the Nextflow plugin framework. ## Timeline @@ -38,7 +38,7 @@ During this time, plugin developers are encouraged to experiment with the Gradle The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. -Developers will be required to publish to the Nextflow plugin registry instead. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. +Developers will be required to publish to the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. ### Nextflow 26.04 @@ -151,7 +151,7 @@ To migrate an existing Nextflow plugin: 7. In the plugin root directory, run `make assemble`. -Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template, add your existing plugin code as needed. See {ref}`dev-plugins-template` for more information about the plugin template. +Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template and add your existing plugin code. See {ref}`dev-plugins-template` for more information about the plugin template. ### Publishing to the Nextflow plugin registry From d011970f4916bc896f579d0ac6464f6c232930e8 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 9 May 2025 21:29:54 -0500 Subject: [PATCH 19/19] Unlist timeline subheadings Signed-off-by: Ben Sherman --- docs/migrating-plugin-registry.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/migrating-plugin-registry.md b/docs/migrating-plugin-registry.md index eacff3a0ed..3d5291365f 100644 --- a/docs/migrating-plugin-registry.md +++ b/docs/migrating-plugin-registry.md @@ -28,21 +28,21 @@ The [legacy plugin index](https://github.com/nextflow-io/plugins) will be deprec The following timeline is tentative and subject to modification. ::: -### Nextflow 25.04 +

Nextflow 25.04

The Nextflow plugin registry is available as a private beta. Nextflow 25.04 can use the Nextflow plugin registry as an opt-in feature. The Nextflow plugin registry will be automatically kept up-to-date with the [legacy plugin index](https://github.com/nextflow-io/plugins). During this time, plugin developers are encouraged to experiment with the Gradle plugin and plugin registry. -### Nextflow 25.10 +

Nextflow 25.10

The Nextflow plugin registry will be generally available. Nextflow 25.10 will use the plugin registry by default. The legacy plugin index will be **closed to new pull requests**. Developers will be required to publish to the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will be automatically kept up-to-date with the Nextflow plugin registry. -### Nextflow 26.04 +

Nextflow 26.04

-Nextflow 25.10 will only be able to use the Nextflow plugin registry. +Nextflow 26.04 will only be able to use the Nextflow plugin registry. At some point in the future, the legacy plugin index will be **frozen** -- it will no longer receives updates from the Nextflow plugin registry. To ensure continued support for older versions of Nextflow, the legacy plugin index will remain available indefinitely.