Enable multiple git revisions (Jun 2024) : final, using commit ID in repo localPath

docs/cli.md

@@ -366,7 +366,7 @@ The `clone` command downloads a pipeline from a Git-hosting platform into the *c
 : Service hub where the project is hosted. Options: `gitlab` or `bitbucket`.
 
 `-r` (`master`)
-: Revision to clone - It can be a git branch, tag, or revision number.
+: Revision of the project to clone (either a git branch, tag or commit SHA number).
 
 `-user`
 : Private repository user name.
@@ -415,6 +415,9 @@ The `config` command is used for printing the project's configuration i.e. the `
 `-properties`
 : Print config using Java properties notation.
 
+`-r, -revision`
+: Revision of the project (either a git branch, tag or commit SHA number).
+
 `-a, -show-profiles`
 : Show all configuration profiles.
 
@@ -535,12 +538,18 @@ The `drop` command is used to remove the projects which have been downloaded int
 
 **Options**
 
+`-a, -all-revisions`
+: For specified project, drop all revisions.
+
 `-f`
 : Delete the repository without taking care of local changes.
 
 `-h, -help`
 : Print the command usage.
 
+`-r, -revision`
+: Revision of the project to drop (either a git branch, tag or commit SHA number).
+
 **Examples**
 
 Drop the `nextflow-io/hello` project.
@@ -664,7 +673,7 @@ $ nextflow info [options] [project]
 
 **Description**
 
-The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project.
+The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project; in this case, note how revisions marked as `P` are pulled locally.
 
 If no run name or session id is provided, it will clean the latest run.
 
@@ -706,10 +715,10 @@ $ nextflow info nextflow-io/hello
   local path  : /Users/evanfloden/.nextflow/assets/nextflow-io/hello
   main script : main.nf
   revisions   :
-  * master (default)
+  P master (default)
     mybranch
     testing
-    v1.1 [t]
+  P v1.1 [t]
     v1.2 [t]
 ```
 
@@ -878,6 +887,12 @@ The `list` commands prints a list of the projects which are already downloaded i
 
 **Options**
 
+`-a, -all-revisions`
+: For each project, also list revisions.
+
+`-d`
+: Show commit information for revisions (in conjunction with `-a`)).
+
 `-h, -help`
 : Print the command usage.
 
@@ -1059,7 +1074,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo
 
 **Options**
 
-`-all`
+`-a, -all`
 : Update all downloaded projects.
 
 `-d, -deep`
@@ -1072,7 +1087,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo
 : Service hub where the project is hosted. Options: `gitlab` or `bitbucket`
 
 `-r, -revision`
-: Revision of the project to run (either a git branch, tag or commit hash).
+: Revision of the project to pull (either a git branch, tag or commit SHA number).
 : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined.
 
 `-user`
@@ -1217,7 +1232,7 @@ The `run` command is used to execute a local pipeline script or remote pipeline
 : Execute the script using the cached results, useful to continue executions that was stopped by an error.
 
 `-r, -revision`
-: Revision of the project to run (either a git branch, tag or commit hash).
+: Revision of the project to run (either a git branch, tag or commit SHA number).
 : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined.
 
 `-stub-run, -stub`
@@ -1426,6 +1441,9 @@ The `view` command is used to inspect the pipelines that are already stored in t
 `-q`
 : Hide header line.
 
+`-r, -revision`
+: Revision of the project (either a git branch, tag or commit SHA number).
+
 **Examples**
 
 Viewing the contents of a downloaded pipeline.

docs/developer/diagrams/nextflow.scm.mmd

@@ -8,9 +8,10 @@ classDiagram
 
     class AssetManager {
         project : String
+        revision : String
         localPath : File
         mainScript : String
-        repositoryProvider : RepositoryProvider
+        provider : RepositoryProvider
         hub : String
         providerConfigs : List~ProviderConfig~
     }

docs/sharing.md

@@ -58,6 +58,15 @@ nextflow run nextflow-io/hello -r v1.1
 
 It will execute two different project revisions corresponding to the Git tag/branch having that names.
 
+:::{versionadded} 24.XX.0-edge
+:::
+
+Nextflow downloads and locally maintains each explicitly requested Git branch, tag or commit ID in a separate directory path, thus enabling to run multiple revisions of the same pipeline at the same time. Each downloaded revision is stored in a subdirecrory path of the local project path: `$NXF_ASSETS/<org>/<repo>/.nextflow/commits/<commitId>`.
+
+:::{warning}
+If you really care about reproducibility of your pipelines, consider explicitly referring to them by tag or commit ID, rather than my branch. This is because the same branch will point to different underlying commits over time, as pipeline development goes on.
+:::
+
 ## Commands to manage projects
 
 The following commands allows you to perform some basic operations that can be used to manage your projects.
@@ -91,16 +100,16 @@ By using the `info` command you can show information from a downloaded project.
 $ nextflow info hello
 project name: nextflow-io/hello
 repository  : http://github.com/nextflow-io/hello
-local path  : $HOME/.nextflow/assets/nextflow-io/hello
+local path  : /Users/evanfloden/.nextflow/assets/nextflow-io/hello
 main script : main.nf
 revisions   :
-* master (default)
+P master (default)
   mybranch
-  v1.1 [t]
+P v1.1 [t]
   v1.2 [t]
 ```
 
-Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local folder where the project has been downloaded; 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the current checked-out revision is marked with a `*` on the left.
+Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local path where the project can be found; 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the locally pulled revisions are marked with a `P` on the left.
 
 ### Pulling or updating a project
 

modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy

@@ -15,6 +15,7 @@
  */
 
 package nextflow.cli
+
 import com.beust.jcommander.Parameter
 import com.beust.jcommander.Parameters
 import groovy.transform.CompileStatic
@@ -37,7 +38,7 @@ class CmdClone extends CmdBase implements HubOptions {
     @Parameter(required=true, description = 'name of the project to clone')
     List<String> args
 
-    @Parameter(names='-r', description = 'Revision to clone - It can be a git branch, tag or revision number')
+    @Parameter(names='-r', description = 'Revision of the project to clone (either a git branch, tag or commit SHA number)')
     String revision
 
     @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth')
@@ -53,6 +54,7 @@ class CmdClone extends CmdBase implements HubOptions {
         // the pipeline name
         String pipeline = args[0]
         final manager = new AssetManager(pipeline, this)
+        manager.setRevisionAndLocalPath(pipeline, revision)
 
         // the target directory is the second parameter
         // otherwise default the current pipeline name
@@ -68,9 +70,9 @@ class CmdClone extends CmdBase implements HubOptions {
         }
 
         manager.checkValidRemoteRepo()
-        print "Cloning ${manager.project}${revision ? ':'+revision:''} ..."
-        manager.clone(target, revision, deep)
+        print "Cloning ${manager.getProjectWithRevision()} ..."
+        manager.clone(target, deep)
         print "\r"
-        println "${manager.project} cloned to: $target"
+        println "${manager.getProjectWithRevision()} cloned to: $target"
     }
 }

modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy

@@ -44,6 +44,9 @@ class CmdConfig extends CmdBase {
     @Parameter(description = 'project name')
     List<String> args = []
 
+    @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)')
+    String revision
+
     @Parameter(names=['-a','-show-profiles'], description = 'Show all configuration profiles')
     boolean showAllProfiles
 
@@ -184,6 +187,7 @@ class CmdConfig extends CmdBase {
         }
 
         final manager = new AssetManager(path)
+        manager.setRevisionAndLocalPath(path, revision)
         manager.isLocal() ? manager.localPath.toPath() : manager.configFile?.parent
 
     }

modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy

@@ -16,6 +16,8 @@
 
 package nextflow.cli
 
+import static nextflow.scm.AssetManager.DEFAULT_REVISION_DIRNAME
+
 import com.beust.jcommander.Parameter
 import com.beust.jcommander.Parameters
 import groovy.transform.CompileStatic
@@ -39,6 +41,12 @@ class CmdDrop extends CmdBase {
     @Parameter(required=true, description = 'name of the project to drop')
     List<String> args
 
+    @Parameter(names=['-r','-revision'], description = 'Revision of the project to drop (either a git branch, tag or commit SHA number)')
+    String revision
+
+    @Parameter(names=['-a','-all-revisions'], description = 'For specified project, drop all revisions')
+    Boolean allRevisions
+
     @Parameter(names='-f', description = 'Delete the repository without taking care of local changes')
     boolean force
 
@@ -48,18 +56,42 @@ class CmdDrop extends CmdBase {
     @Override
     void run() {
         Plugins.init()
-        def manager = new AssetManager(args[0])
-        if( !manager.localPath.exists() ) {
-            throw new AbortOperationException("No match found for: ${args[0]}")
+
+        List<AssetManager> dropList = []
+        if ( allRevisions ) {
+            def revManager = new AssetManager(args[0])
+            revManager.listRevisions().each { rev ->
+                if( rev == DEFAULT_REVISION_DIRNAME )
+                    rev = null
+                dropList << new AssetManager(args[0]).setRevisionAndLocalPath(args[0], rev)
+            }
+        } else {
+            dropList << new AssetManager(args[0]).setRevisionAndLocalPath(args[0], revision)
         }
 
-        if( this.force || manager.isClean() ) {
-            manager.close()
-            if( !manager.localPath.deleteDir() )
-                throw new AbortOperationException("Unable to delete project `${manager.project}` -- Check access permissions for path: ${manager.localPath}")
-            return
+        if ( !dropList ) {
+            throw new AbortOperationException("No revisions found for specified project: ${args[0]}")
         }
 
-        throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it")
+        dropList.each { manager ->
+            if( !manager.localPathDefinedAndExists() ) {
+                throw new AbortOperationException("No match found for: ${manager.getProjectWithRevision()}")
+            }
+
+            if( this.force || manager.isClean() ) {
+                manager.close()
+                if( !manager.localPath.deleteDir() )
+                    throw new AbortOperationException("Unable to delete project `${manager.getProjectWithRevision()}` -- Check access permissions for path: ${manager.localPath}")
+                manager.pruneRevisionMap(manager.revision)
+                return
+            }
+
+            throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it")
+        }
+
+        if ( allRevisions ) {
+            def revManager = new AssetManager(args[0])
+            revManager.localRootPath.deleteDir()
+        }
     }
 }

modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy

@@ -75,9 +75,20 @@ class CmdInfo extends CmdBase {
         }
 
         Plugins.init()
-        final manager = new AssetManager(args[0])
-        if( !manager.isLocal() )
-            throw new AbortOperationException("Unknown project `${args[0]}`")
+        def manager = new AssetManager(args[0])
+        manager.setRevisionAndLocalPath(args[0], null)
+        if( !manager.isLocal() ) {
+            // if default branch not found locally, use first one from list of local pulls
+            if ( manager.listRevisions() ) {
+                manager = new AssetManager(args[0])
+                manager.setRevisionAndLocalPath(args[0], manager.listRevisions()[0])
+                if( !manager.isLocal() )
+                    throw new AbortOperationException("Unknown project `${args[0]}`")
+            }
+            else {
+                throw new AbortOperationException("Unknown project `${args[0]}`")
+            }
+        }
 
         if( !format || format == 'text' ) {
             printText(manager,level)
@@ -101,7 +112,7 @@ class CmdInfo extends CmdBase {
 
         out.println " project name: ${manager.project}"
         out.println " repository  : ${manager.repositoryUrl}"
-        out.println " local path  : ${manager.localPath}"
+        out.println " local path  : ${manager.localRootPath.toString()}"
         out.println " main script : ${manager.mainScriptName}"
         if( manager.homePage && manager.homePage != manager.repositoryUrl )
             out.println " home page   : ${manager.homePage}"
@@ -138,7 +149,7 @@ class CmdInfo extends CmdBase {
         def result = [:]
         result.projectName = manager.project
         result.repository = manager.repositoryUrl
-        result.localPath = manager.localPath?.toString()
+        result.localPath = manager.localRootPath.toString()
         result.manifest = manager.manifest.toMap()
         result.revisions = manager.getBranchesAndTags(checkForUpdates)
         return result