how to load a Nextflow Plugin from a private git repo

[stevekm]

in the Nextflow Plugin docs ( https://www.nextflow.io/docs/latest/plugins.html#using-plugins ) it describes config syntax like this to specify a version of a plugin

plugins {
    id 'nf-hello@0.1.0'
}

Further down, the "Testing" section describes how to use a custom registry for plugins ( https://www.nextflow.io/docs/latest/plugins.html#testing-plugins )

# custom registry at https://github.com/my-org/plugins
export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json"

# custom plugin release
export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json"

nextflow run <pipeline> -plugins nf-hello

It seems that the "registry" here contains a list of all available Nextflow Plugins; https://github.com/nextflow-io/plugins/blob/main/plugins.json

Its not clear to me how we can combine these methods, such that we can still use the default Nextflow Plugins such as nf-prov from the official sources, and also include some custom modules that may be hosted internally on a private git repository.

Based on these docs, it sounds as if you would have to mirror the entire official registry JSON yourself, and add in the URL to your custom plugin. Its not clear how credentials for the custom plugin would be passed.

I think it would be a lot easier if something like the syntax from pip could be used, it looks like this;

(from conda environment.yml)

  - pip:
   - git+https://github.com/myorg/my-plugin-repo.git@1.2.3

something like this which lets you bundle both a git repo URL and a version label along with the plugin id might be helpful, is this possible? Are there other methods available to allow for providing the path directly to the plugin to use, such as maybe a S3 path to a .jar file in a bucket somewhere?

[bentsherman]

Nextflow uses NXF_PLUGINS_TEST_REPOSITORY in conjunction with the public registry. Also, the environment var can be a comma-separated list of registries / repositories, not sure if that is documented

I'm not sure how the authentication works, i.e. if the plugin manager uses the GIt config when accessing a private plugin registry, might be something to look into

[stevekm]

so if I specify an extra registry via the env var, it will not exclude the usage of the official registry?

also, calling the var NXF_PLUGINS_TEST_REPOSITORY does not give confidence when the desired goal is to use the private plugins repo in production, not for testing.

[bentsherman]

so if I specify an extra registry via the env var, it will not exclude the usage of the official registry?

correct

also, calling the var NXF_PLUGINS_TEST_REPOSITORY does not give confidence when the desired goal is to use the private plugins repo in production, not for testing.

the original purpose was for testing plugins, but it happens to cover the private registry use case. likely something we will review when we build out a proper registry service for plugins to replace the JSON index file

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.