diff --git a/docs/cli.md b/docs/cli.md index df93409d7d..e7fc50955b 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -366,7 +366,7 @@ The `clone` command downloads a pipeline from a Git-hosting platform into the *c : Service hub where the project is hosted. Options: `gitlab` or `bitbucket`. `-r` (`master`) -: Revision to clone - It can be a git branch, tag, or revision number. +: Revision of the project to clone (either a git branch, tag or commit SHA number). `-user` : Private repository user name. @@ -415,6 +415,9 @@ The `config` command is used for printing the project's configuration i.e. the ` `-properties` : Print config using Java properties notation. +`-r, -revision` +: Revision of the project (either a git branch, tag or commit SHA number). + `-a, -show-profiles` : Show all configuration profiles. @@ -535,12 +538,18 @@ The `drop` command is used to remove the projects which have been downloaded int **Options** +`-a, -all-revisions` +: For specified project, drop all revisions. + `-f` : Delete the repository without taking care of local changes. `-h, -help` : Print the command usage. +`-r, -revision` +: Revision of the project to drop (either a git branch, tag or commit SHA number). + **Examples** Drop the `nextflow-io/hello` project. @@ -664,7 +673,7 @@ $ nextflow info [options] [project] **Description** -The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project. +The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project; in this case, note how revisions marked as `P` are pulled locally. If no run name or session id is provided, it will clean the latest run. @@ -706,10 +715,10 @@ $ nextflow info nextflow-io/hello local path : /Users/evanfloden/.nextflow/assets/nextflow-io/hello main script : main.nf revisions : - * master (default) + P master (default) mybranch testing - v1.1 [t] + P v1.1 [t] v1.2 [t] ``` @@ -878,6 +887,12 @@ The `list` commands prints a list of the projects which are already downloaded i **Options** +`-a, -all-revisions` +: For each project, also list revisions. + +`-d` +: Show commit information for revisions (in conjunction with `-a`)). + `-h, -help` : Print the command usage. @@ -1059,7 +1074,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo **Options** -`-all` +`-a, -all` : Update all downloaded projects. `-d, -deep` @@ -1072,7 +1087,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo : Service hub where the project is hosted. Options: `gitlab` or `bitbucket` `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Revision of the project to pull (either a git branch, tag or commit SHA number). : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-user` @@ -1217,7 +1232,7 @@ The `run` command is used to execute a local pipeline script or remote pipeline : Execute the script using the cached results, useful to continue executions that was stopped by an error. `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Revision of the project to run (either a git branch, tag or commit SHA number). : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-stub-run, -stub` @@ -1426,6 +1441,9 @@ The `view` command is used to inspect the pipelines that are already stored in t `-q` : Hide header line. +`-r, -revision` +: Revision of the project (either a git branch, tag or commit SHA number). + **Examples** Viewing the contents of a downloaded pipeline. diff --git a/docs/developer/diagrams/nextflow.scm.mmd b/docs/developer/diagrams/nextflow.scm.mmd index 1c52075372..5d015138b4 100644 --- a/docs/developer/diagrams/nextflow.scm.mmd +++ b/docs/developer/diagrams/nextflow.scm.mmd @@ -8,9 +8,10 @@ classDiagram class AssetManager { project : String + revision : String localPath : File mainScript : String - repositoryProvider : RepositoryProvider + provider : RepositoryProvider hub : String providerConfigs : List~ProviderConfig~ } diff --git a/docs/sharing.md b/docs/sharing.md index 2db5b9b527..91308229af 100644 --- a/docs/sharing.md +++ b/docs/sharing.md @@ -58,6 +58,15 @@ nextflow run nextflow-io/hello -r v1.1 It will execute two different project revisions corresponding to the Git tag/branch having that names. +:::{versionadded} 24.XX.0-edge +::: + +Nextflow downloads and locally maintains each explicitly requested Git branch, tag or commit ID in a separate directory path, thus enabling to run multiple revisions of the same pipeline at the same time. Each downloaded revision is stored in a subdirecrory path of the local project path: `$NXF_ASSETS///.nextflow/commits/`. + +:::{warning} +If you really care about reproducibility of your pipelines, consider explicitly referring to them by tag or commit ID, rather than my branch. This is because the same branch will point to different underlying commits over time, as pipeline development goes on. +::: + ## Commands to manage projects The following commands allows you to perform some basic operations that can be used to manage your projects. @@ -91,16 +100,16 @@ By using the `info` command you can show information from a downloaded project. $ nextflow info hello project name: nextflow-io/hello repository : http://github.com/nextflow-io/hello -local path : $HOME/.nextflow/assets/nextflow-io/hello +local path : /Users/evanfloden/.nextflow/assets/nextflow-io/hello main script : main.nf revisions : -* master (default) +P master (default) mybranch - v1.1 [t] +P v1.1 [t] v1.2 [t] ``` -Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local folder where the project has been downloaded; 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the current checked-out revision is marked with a `*` on the left. +Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local path where the project can be found; 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the locally pulled revisions are marked with a `P` on the left. ### Pulling or updating a project diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy index 9c582aaaf6..9fd1b6ad37 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy @@ -15,6 +15,7 @@ */ package nextflow.cli + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -37,7 +38,7 @@ class CmdClone extends CmdBase implements HubOptions { @Parameter(required=true, description = 'name of the project to clone') List args - @Parameter(names='-r', description = 'Revision to clone - It can be a git branch, tag or revision number') + @Parameter(names='-r', description = 'Revision of the project to clone (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') @@ -53,6 +54,7 @@ class CmdClone extends CmdBase implements HubOptions { // the pipeline name String pipeline = args[0] final manager = new AssetManager(pipeline, this) + manager.setRevisionAndLocalPath(pipeline, revision) // the target directory is the second parameter // otherwise default the current pipeline name @@ -68,9 +70,9 @@ class CmdClone extends CmdBase implements HubOptions { } manager.checkValidRemoteRepo() - print "Cloning ${manager.project}${revision ? ':'+revision:''} ..." - manager.clone(target, revision, deep) + print "Cloning ${manager.getProjectWithRevision()} ..." + manager.clone(target, deep) print "\r" - println "${manager.project} cloned to: $target" + println "${manager.getProjectWithRevision()} cloned to: $target" } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy index 5ff2156b18..30b5124edf 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy @@ -44,6 +44,9 @@ class CmdConfig extends CmdBase { @Parameter(description = 'project name') List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names=['-a','-show-profiles'], description = 'Show all configuration profiles') boolean showAllProfiles @@ -184,6 +187,7 @@ class CmdConfig extends CmdBase { } final manager = new AssetManager(path) + manager.setRevisionAndLocalPath(path, revision) manager.isLocal() ? manager.localPath.toPath() : manager.configFile?.parent } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy index 9d67190a54..311738ad5b 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.DEFAULT_REVISION_DIRNAME + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -39,6 +41,12 @@ class CmdDrop extends CmdBase { @Parameter(required=true, description = 'name of the project to drop') List args + @Parameter(names=['-r','-revision'], description = 'Revision of the project to drop (either a git branch, tag or commit SHA number)') + String revision + + @Parameter(names=['-a','-all-revisions'], description = 'For specified project, drop all revisions') + Boolean allRevisions + @Parameter(names='-f', description = 'Delete the repository without taking care of local changes') boolean force @@ -48,18 +56,42 @@ class CmdDrop extends CmdBase { @Override void run() { Plugins.init() - def manager = new AssetManager(args[0]) - if( !manager.localPath.exists() ) { - throw new AbortOperationException("No match found for: ${args[0]}") + + List dropList = [] + if ( allRevisions ) { + def revManager = new AssetManager(args[0]) + revManager.listRevisions().each { rev -> + if( rev == DEFAULT_REVISION_DIRNAME ) + rev = null + dropList << new AssetManager(args[0]).setRevisionAndLocalPath(args[0], rev) + } + } else { + dropList << new AssetManager(args[0]).setRevisionAndLocalPath(args[0], revision) } - if( this.force || manager.isClean() ) { - manager.close() - if( !manager.localPath.deleteDir() ) - throw new AbortOperationException("Unable to delete project `${manager.project}` -- Check access permissions for path: ${manager.localPath}") - return + if ( !dropList ) { + throw new AbortOperationException("No revisions found for specified project: ${args[0]}") } - throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + dropList.each { manager -> + if( !manager.localPathDefinedAndExists() ) { + throw new AbortOperationException("No match found for: ${manager.getProjectWithRevision()}") + } + + if( this.force || manager.isClean() ) { + manager.close() + if( !manager.localPath.deleteDir() ) + throw new AbortOperationException("Unable to delete project `${manager.getProjectWithRevision()}` -- Check access permissions for path: ${manager.localPath}") + manager.pruneRevisionMap(manager.revision) + return + } + + throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + } + + if ( allRevisions ) { + def revManager = new AssetManager(args[0]) + revManager.localRootPath.deleteDir() + } } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy index c8c1c31bc0..b02a367220 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy @@ -75,9 +75,20 @@ class CmdInfo extends CmdBase { } Plugins.init() - final manager = new AssetManager(args[0]) - if( !manager.isLocal() ) - throw new AbortOperationException("Unknown project `${args[0]}`") + def manager = new AssetManager(args[0]) + manager.setRevisionAndLocalPath(args[0], null) + if( !manager.isLocal() ) { + // if default branch not found locally, use first one from list of local pulls + if ( manager.listRevisions() ) { + manager = new AssetManager(args[0]) + manager.setRevisionAndLocalPath(args[0], manager.listRevisions()[0]) + if( !manager.isLocal() ) + throw new AbortOperationException("Unknown project `${args[0]}`") + } + else { + throw new AbortOperationException("Unknown project `${args[0]}`") + } + } if( !format || format == 'text' ) { printText(manager,level) @@ -101,7 +112,7 @@ class CmdInfo extends CmdBase { out.println " project name: ${manager.project}" out.println " repository : ${manager.repositoryUrl}" - out.println " local path : ${manager.localPath}" + out.println " local path : ${manager.localRootPath.toString()}" out.println " main script : ${manager.mainScriptName}" if( manager.homePage && manager.homePage != manager.repositoryUrl ) out.println " home page : ${manager.homePage}" @@ -138,7 +149,7 @@ class CmdInfo extends CmdBase { def result = [:] result.projectName = manager.project result.repository = manager.repositoryUrl - result.localPath = manager.localPath?.toString() + result.localPath = manager.localRootPath.toString() result.manifest = manager.manifest.toMap() result.revisions = manager.getBranchesAndTags(checkForUpdates) return result diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy index 5b08be249a..261ff10698 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy @@ -16,6 +16,9 @@ package nextflow.cli +import static nextflow.scm.AssetManager.DEFAULT_REVISION_DIRNAME + +import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic import groovy.util.logging.Slf4j @@ -33,6 +36,18 @@ class CmdList extends CmdBase { static final public NAME = 'list' + @Parameter(names=['-a','-all-revisions'], description = 'For each project, also list revisions') + Boolean allRevisions + + @Parameter(names=['-all-commits'], description = 'For each project, also list all downloaded commits') + Boolean allCommits + + @Parameter(names='-d',description = 'Show commit information for revisions', arity = 0) + boolean detailed + + @Parameter(names='-dd', hidden = true, arity = 0) + boolean moreDetailed + @Override final String getName() { NAME } @@ -45,7 +60,39 @@ class CmdList extends CmdBase { return } - all.each { println it } + if( moreDetailed ) + detailed = true + if( detailed && allRevisions ) { + all.each{ + println(" $it") + def revManager = new AssetManager(it) + revManager.listRevisionsAndCommits().each{ k,v -> + if( k == DEFAULT_REVISION_DIRNAME ) + k = '(default)' + if( !moreDetailed ) + v = v.substring(0,10) + println(" $v $k") } + } + } + else if( allRevisions ) { + all.each{ + println(" $it") + def revManager = new AssetManager(it) + revManager.listRevisions().each{ + if( it == DEFAULT_REVISION_DIRNAME ) + it = '(default)' + println(" $it") + } + } + } else if( allCommits ) { + all.each{ + println(" $it") + def revManager = new AssetManager(it) + revManager.listCommits().each{ println(" $it") } + } + } else { + all.each{ println(" $it") } + } } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy index 9d2fa9a01d..cf5c661a56 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy @@ -15,6 +15,9 @@ */ package nextflow.cli + +import static nextflow.scm.AssetManager.DEFAULT_REVISION_DIRNAME + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -37,10 +40,10 @@ class CmdPull extends CmdBase implements HubOptions { @Parameter(description = 'project name or repository url to pull', arity = 1) List args - @Parameter(names='-all', description = 'Update all downloaded projects', arity = 0) + @Parameter(names=['-a','-all'], description = 'Update all downloaded projects', arity = 0) boolean all - @Parameter(names=['-r','-revision'], description = 'Revision of the project to run (either a git branch, tag or commit SHA number)') + @Parameter(names=['-r','-revision'], description = 'Revision of the project to pull (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') @@ -58,11 +61,11 @@ class CmdPull extends CmdBase implements HubOptions { if( !all && !args ) throw new AbortOperationException('Missing argument') - def list = all ? AssetManager.list() : args.toList() - if( !list ) { - log.info "(nothing to do)" - return - } + if( all && args ) + throw new AbortOperationException('Option `all` requires no arguments') + + if( all && revision ) + throw new AbortOperationException('Option `all` is not compatible with `revision`') /* only for testing purpose */ if( root ) { @@ -71,12 +74,33 @@ class CmdPull extends CmdBase implements HubOptions { // init plugin system Plugins.init() - - list.each { - log.info "Checking $it ..." - def manager = new AssetManager(it, this) - def result = manager.download(revision,deep) + List list = [] + if ( all ) { + def all = AssetManager.list() + all.each{ proj -> + def revManager = new AssetManager(proj) + revManager.listRevisions().each{ rev -> + if( rev == DEFAULT_REVISION_DIRNAME ) + rev = null + list << new AssetManager(proj, this).setRevisionAndLocalPath(proj, rev) + } + } + } else { + args.toList().each { + list << new AssetManager(it, this).setRevisionAndLocalPath(it, revision) + } + } + + if( !list ) { + log.info "(nothing to do)" + return + } + + list.each { manager -> + log.info "Checking ${manager.getProjectWithRevision()} ..." + + def result = manager.download(deep) manager.updateModules() def scriptFile = manager.getScriptFile() diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy index e5d18bdb87..36e56bb21f 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy @@ -570,21 +570,21 @@ class CmdRun extends CmdBase implements HubOptions { * try to look for a pipeline in the repository */ def manager = new AssetManager(pipelineName, this) - def repo = manager.getProject() + manager.setRevisionAndLocalPath(pipelineName, revision) + def repo = manager.getProjectWithRevision() boolean checkForUpdate = true if( !manager.isRunnable() || latest ) { if( offline ) throw new AbortOperationException("Unknown project `$repo` -- NOTE: automatic download from remote repositories is disabled") log.info "Pulling $repo ..." - def result = manager.download(revision,deep) + def result = manager.download(deep) if( result ) log.info " $result" checkForUpdate = false } - // checkout requested revision + // post download operations try { - manager.checkout(revision) manager.updateModules() final scriptFile = manager.getScriptFile(mainScript) if( checkForUpdate && !offline ) diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy index dcdcfcae67..96eaa22b39 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy @@ -42,6 +42,9 @@ class CmdView extends CmdBase { @Parameter(description = 'project name', required = true) List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names = '-q', description = 'Hide header line', arity = 0) boolean quiet @@ -52,8 +55,9 @@ class CmdView extends CmdBase { void run() { Plugins.init() def manager = new AssetManager(args[0]) + manager.setRevisionAndLocalPath(args[0], revision) if( !manager.isLocal() ) - throw new AbortOperationException("Unknown project name `${args[0]}`") + throw new AbortOperationException("Unknown project `${manager.getProjectWithRevision()}`") if( all ) { if( !quiet ) diff --git a/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy b/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy index c50cef627d..b1f9e83d92 100644 --- a/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy @@ -274,6 +274,7 @@ class K8sDriverLauncher { // -- check and parse project remote config Plugins.init() final pipelineConfig = new AssetManager(pipelineName, cmd) .getConfigFile() + pipelineConfig.setRevisionAndLocalPath(pipelineName, cmd.revision) builder.setUserConfigFiles(pipelineConfig) } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy index dd304ad59e..acd7a2d898 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy @@ -62,6 +62,14 @@ class AssetManager { @PackageScope static File root = DEFAULT_ROOT + static final String BARE_REPO = '.nextflow/bare_repo' + + static final String REVISION_MAP = '.nextflow/revision_map_file' + + static public final String REVISION_SUBDIR = '.nextflow/commits' + + static public final String DEFAULT_REVISION_DIRNAME = 'DEFAULT_REVISION' + /** * The pipeline name. It must be in the form {@code username/repo} where 'username' * is a valid user name or organisation account, while 'repo' is the repository name @@ -69,13 +77,23 @@ class AssetManager { */ private String project + /** + * The name of the commit/branch/tag as requested via command line + * This is now a first class attribute of a pipeline + */ + private String revision + /** * Directory where the pipeline is cloned (i.e. downloaded) + * + * Schema: $NXF_ASSETS///.nextflow/commits/ */ private File localPath private Git _git + private Git _bareGit + private String mainScript private RepositoryProvider provider @@ -94,7 +112,7 @@ class AssetManager { /** * Create a new asset manager with the specified pipeline name * - * @param pipeline The pipeline to be managed by this manager e.g. {@code nextflow-io/hello} + * @param pipelineName The pipeline to be managed by this manager e.g. {@code nextflow-io/hello} */ AssetManager( String pipelineName, HubOptions cliOpts = null) { assert pipelineName @@ -104,7 +122,7 @@ class AssetManager { build(pipelineName, config, cliOpts) } - AssetManager( String pipelineName, Map config ) { + AssetManager( String pipelineName, Map config) { assert pipelineName // build the object build(pipelineName, config) @@ -124,18 +142,45 @@ class AssetManager { this.providerConfigs = ProviderConfig.createFromMap(config) this.project = resolveName(pipelineName) - this.localPath = checkProjectDir(project) + validateProjectName(this.project) + this.hub = checkHubProvider(cliOpts) this.provider = createHubProvider(hub) setupCredentials(cliOpts) - validateProjectDir() + validateProjectBareDir() return this } @PackageScope File getLocalGitConfig() { - localPath ? new File(localPath,'.git/config') : null + bareRepo.exists() ? new File(bareRepo,'config') : null + } + + @PackageScope + File getBareRepo() { + new File(root, project + '/' + BARE_REPO) + } + + /** + * Path of the "revision -> commit" map file for the project + * + * CSV format: , + * + * Schema: $NXF_ASSETS///.nextflow/revisionMapFile + */ + @PackageScope + File getRevisionMap() { + new File(root, project + '/' + REVISION_MAP) + } + + @PackageScope + File getRevisionSubdir( String projectName = project ) { + new File(root, projectName + '/' + REVISION_SUBDIR) + } + + File getLocalRootPath() { + new File(root, project) } @PackageScope AssetManager setProject(String name) { @@ -169,20 +214,29 @@ class AssetManager { } /** - * Verify the project name matcher the expected pattern. - * and return the directory where the project is stored locally + * Verify the project name matches the expected pattern + */ + @PackageScope + void validateProjectName(String projectName) { + if( !isValidProjectName(projectName)) { + throw new IllegalArgumentException("Not a valid project name: $projectName") + } + } + + /** + * Map revision to commit ID, + * define the directory where the project is stored locally, + * and validate it. * * @param projectName A project name matching the pattern {@code owner/project} + * @param revision Revision ID for the selected pipeline (git branch, tag or commit SHA number) * @return The project dir {@link File} */ @PackageScope - File checkProjectDir(String projectName) { - - if( !isValidProjectName(projectName)) { - throw new IllegalArgumentException("Not a valid project name: $projectName") + void updateProjectDir(String projectName, String commitId) { + if( commitId ) { + this.localPath = new File( root, projectName + '/' + REVISION_SUBDIR + '/' + commitId ) } - - new File(root, project) } /** @@ -190,24 +244,125 @@ class AssetManager { * line option or implicitly by entering a repository URL, matches with clone URL of a project already cloned (downloaded). */ @PackageScope - void validateProjectDir() { - - if( !localPath.exists() ) { + void validateProjectBareDir() { + if( !bareRepo.exists() ) { return } // if project dir exists it must contain the Git config file final configProvider = guessHubProviderFromGitConfig(true) if( !configProvider ) - throw new IllegalStateException("Cannot find a provider config for repository at path: $localPath") + throw new IllegalStateException("Cannot find a provider config for repository at path: $bareRepo") // checks that the selected hub matches with the one defined in the git config file if( hub != configProvider ) { - throw new AbortOperationException("A project with name: `$localPath` has already been downloaded from a different provider: `$configProvider`") + throw new AbortOperationException("A project with name: `$bareRepo` has already been downloaded from a different provider: `$configProvider`") } } + @PackageScope + void checkBareRepo() { + /* + * if the bare repository of the pipeline does not exists locally pull it from the remote repo + */ + if( !bareRepo.exists() ) { + bareRepo.parentFile.mkdirs() + + final cloneURL = getGitRepositoryUrl() + log.debug "Pulling bare repo for $project -- Using remote clone url: ${cloneURL}" + + def bare = Git.cloneRepository() + if( provider.hasCredentials() ) + bare.setCredentialsProvider( provider.getGitCredentials() ) + + bare + .setBare( true ) + .setURI(cloneURL) + .setGitDir(bareRepo) + .call() + } else { + log.debug "Fetching (updating) bare repo for $project" + + Git.open(bareRepo) + .fetch() + .call() + } + } + + @PackageScope + String revisionToCommitWithBareRepo(String revision) { + String commitId + + if( bareRepo.exists() ) { + def rev = Git.open(bareRepo) + .getRepository() + .resolve(revision ?: Constants.HEAD) + if( rev ) + commitId = rev.getName() + } + + return commitId + } + + @PackageScope + String revisionToCommitWithMap(String revision) { + String commitId + + if( revisionMap.exists() ) { + String revisionTmp = revision ?: DEFAULT_REVISION_DIRNAME + commitId = revisionMap.readLines().find{ it.split(',')[0] == revisionTmp } + commitId = commitId ? commitId.split(',')[1] : commitId + } + + return commitId + } + + void pruneRevisionMap(String revision) { + if( revisionMap.exists() ) { + String commitId = revisionToCommitWithMap(revision) + List oldRevisionMap = revisionMap.readLines() + revisionMap.text = '' + oldRevisionMap.each{ + if( it.split(',')[1] != commitId ) + revisionMap << it + '\n' + } + } + } + + @PackageScope + void updateRevisionMap(String revision, String commitId) { + String revisionTmp = revision ?: DEFAULT_REVISION_DIRNAME + if( !revisionMap.exists() ) { + revisionMap.parentFile.mkdirs() + revisionMap << revisionTmp + ',' + commitId + '\n' + } else { + List oldRevisionMap = revisionMap.readLines() + revisionMap.text = '' + oldRevisionMap.each{ + if( it.split(',')[0] != revisionTmp ) + revisionMap << it + '\n' + } + revisionMap << revisionTmp + ',' + commitId + '\n' + } + } + + AssetManager setRevisionAndLocalPath(String pipelineName, String revision) { + this.revision = revision + updateProjectDir(resolveName(pipelineName), revisionToCommitWithMap(revision)) + + return this + } + + @PackageScope + void updateRevisionMapAndLocalPath(String revision) { + + String commitId = revisionToCommitWithBareRepo(revision) + + updateRevisionMap(revision, commitId) + updateProjectDir(this.project, commitId) + } + /** * Find out the "hub provider" (i.e. the platform on which the remote repository is stored * for example: github, bitbucket, etc) and verifies that it is a known provider. @@ -300,6 +455,10 @@ class AssetManager { String getProject() { project } + String getRevision() { revision } + + String getProjectWithRevision() { project + ( revision ? ':' + revision : '' ) } + String getHub() { hub } @PackageScope @@ -363,7 +522,7 @@ class AssetManager { return this } - AssetManager checkValidRemoteRepo(String revision=null) { + AssetManager checkValidRemoteRepo() { // Configure the git provider to use the required revision as source for all needed remote resources: // - config if present in repo (nextflow.config by default) // - main script (main.nf by default) @@ -375,11 +534,6 @@ class AssetManager { @Memoized String getGitRepositoryUrl() { - - if( localPath.exists() ) { - return localPath.toURI().toString() - } - provider.getCloneUrl() } @@ -434,7 +588,7 @@ class AssetManager { String text = null ConfigObject result = null try { - text = localPath.exists() ? new File(localPath, MANIFEST_FILE_NAME).text : provider.readText(MANIFEST_FILE_NAME) + text = localPathDefinedAndExists() ? new File(localPath, MANIFEST_FILE_NAME).text : provider.readText(MANIFEST_FILE_NAME) } catch( FileNotFoundException e ) { log.debug "Project manifest does not exist: ${e.message}" @@ -459,7 +613,7 @@ class AssetManager { } Path getConfigFile() { - if( localPath.exists() ) { + if( localPathDefinedAndExists() ) { return new File(localPath, MANIFEST_FILE_NAME).toPath() } else { @@ -485,7 +639,7 @@ class AssetManager { } boolean isLocal() { - localPath.exists() + localPathDefinedAndExists() } /** @@ -493,7 +647,11 @@ class AssetManager { * file (i.e. main.nf) or the nextflow manifest file (i.e. nextflow.config) */ boolean isRunnable() { - localPath.exists() && ( new File(localPath,DEFAULT_MAIN_FILE_NAME).exists() || new File(localPath,MANIFEST_FILE_NAME).exists() ) + localPathDefinedAndExists() && ( new File(localPath,DEFAULT_MAIN_FILE_NAME).exists() || new File(localPath,MANIFEST_FILE_NAME).exists() ) + } + + boolean localPathDefinedAndExists() { + localPath != null ? localPath.exists() : false } /** @@ -517,6 +675,10 @@ class AssetManager { _git.close() _git = null } + if( _bareGit ) { + _bareGit.close() + _bareGit = null + } } /** @@ -538,6 +700,66 @@ class AssetManager { return result } + /** + * @return The list of available revisions for a given project name + */ + List listRevisions( String projectName = this.project ) { + log.debug "Listing revisions for project: $projectName" + + def result = new LinkedList() + if( !root.exists() ) + return result + if( !revisionMap.exists() ) + return result + + revisionMap.eachLine{ it -> result << it.split(',')[0] } + + return result + } + + /** + * uses getDefaultBranch() + * only for usage within service methods getRevisions() and getBranchesAndTags() + */ + @PackageScope + List listRevisionsToCompareInfo() { + listRevisions().collect{ it = ( it != DEFAULT_REVISION_DIRNAME ? it : getDefaultBranch() ) } + } + + /** + * @return The map of available revisions and corresponding commits for a given project name + */ + Map listRevisionsAndCommits( String projectName = this.project ) { + log.debug "Listing revisions for project: $projectName" + + def result = new LinkedHashMap() + if( !root.exists() ) + return result + if( !revisionMap.exists() ) + return result + + revisionMap.eachLine{ it -> result[ it.split(',')[0] ] = it.split(',')[1] } + + return result + } + + /** + * @return The list of downloaded bare commits for a given project name + */ + List listCommits( String projectName = this.project ) { + log.debug "Listing all commits for project: $projectName" + + def result = new LinkedList() + if( !root.exists() ) + return result + if( !getRevisionSubdir(projectName).exists() ) + return result + + getRevisionSubdir(projectName).eachDir { File it -> result << it.getName().toString() } + + return result + } + static protected def find( String name ) { def exact = [] def partial = [] @@ -562,24 +784,38 @@ class AssetManager { return _git } + protected Git getBareGit() { + if( !_bareGit ) { + _bareGit = Git.open(bareRepo) + } + return _bareGit + } + /** * Download a pipeline from a remote Github repository * - * @param revision The revision to download * @result A message representing the operation result */ - String download(String revision=null, Integer deep=null) { + String download(Integer deep=null, boolean testDisableUpdateLocalPath = false) { assert project + // make sure it contains a valid repository + checkValidRemoteRepo() + + // get local copy of bare repository + checkBareRepo() + // update mapping of revision to commit, and update localPath + // boolean is for testing purposes only (e.g. UpdateModuleTest) + if( !testDisableUpdateLocalPath ) + updateRevisionMapAndLocalPath(revision) + /* - * if the pipeline already exists locally pull it from the remote repo + * if the pipeline does not exists locally pull it from the remote repo */ if( !localPath.exists() ) { localPath.parentFile.mkdirs() - // make sure it contains a valid repository - checkValidRemoteRepo(revision) - final cloneURL = getGitRepositoryUrl() + final cloneURL = bareRepo.toString() log.debug "Pulling $project -- Using remote clone url: ${cloneURL}" // clone it @@ -599,14 +835,14 @@ class AssetManager { // use an explicit checkout command *after* the clone instead of cloning a specific branch // because the clone command does not allow the use of SHA commit id (only branch and tag names) try { git.checkout() .setName(revision) .call() } - catch ( RefNotFoundException e ) { checkoutRemoteBranch(revision) } + catch ( RefNotFoundException e ) { checkoutRemoteBranch() } } // return status message - return "downloaded from ${cloneURL}" + return "downloaded from ${getGitRepositoryUrl()}" } - log.debug "Pull pipeline $project -- Using local path: $localPath" + log.debug "Pulling $project -- Using local path: $localPath" // verify that is clean if( !isClean() ) @@ -624,7 +860,7 @@ class AssetManager { * Try to checkout it from a remote branch and return */ catch ( RefNotFoundException e ) { - final ref = checkoutRemoteBranch(revision) + final ref = checkoutRemoteBranch() final commitId = ref?.getObjectId() return commitId ? "checked out at ${commitId.name()}" @@ -662,13 +898,12 @@ class AssetManager { * Clone a pipeline from a remote pipeline repository to the specified folder * * @param directory The folder when the pipeline will be cloned - * @param revision The revision to be cloned. It can be a branch, tag, or git revision number */ - void clone(File directory, String revision = null, Integer deep=null) { + void clone(File directory, Integer deep=null) { def clone = Git.cloneRepository() def uri = getGitRepositoryUrl() - log.debug "Clone project `$project` -- Using remote URI: ${uri} into: $directory" + log.debug "Cloning `${project}` from ${uri} into ${directory}" if( !uri ) throw new AbortOperationException("Cannot find the specified project: $project") @@ -733,30 +968,31 @@ class AssetManager { /** * @return A list of existing branches and tags names. For example *
-     *     * master (default)
-     *       patch-x
-     *       v1.0 (t)
-     *       v1.1 (t)
+     *     * P master (default)
+     *         patch-x
+     *         v1.0 (t)
+     *         v1.1 (t)
      * 
* - * The star character on the left highlight the current revision, the string {@code (default)} - * ticks that it is the default working branch (usually the master branch), while the string {@code (t)} - * shows that the revision is a git tag (instead of a branch) + * The character {@code P} on the left indicates the revision is pulled locally, + * the string {@code (default)} ticks that it is the default working branch, + * while the string {@code (t)} shows that the revision is a git tag (instead of a branch) */ @Deprecated List getRevisions(int level) { def current = getCurrentRevision() def master = getDefaultBranch() + def pulled = listRevisionsToCompareInfo() List branches = getBranchList() .findAll { it.name.startsWith('refs/heads/') || it.name.startsWith('refs/remotes/origin/') } .unique { shortenRefName(it.name) } - .collect { Ref it -> refToString(it,current,master,false,level) } + .collect { Ref it -> refToString(it,current,master,pulled,false,level) } List tags = getTagList() .findAll { it.name.startsWith('refs/tags/') } - .collect { refToString(it,current,master,true,level) } + .collect { refToString(it,current,master,pulled,true,level) } def result = new ArrayList(branches.size() + tags.size()) result.addAll(branches) @@ -777,25 +1013,24 @@ class AssetManager { Map getBranchesAndTags(boolean checkForUpdates) { final result = [:] - final current = getCurrentRevision() final master = getDefaultBranch() final branches = [] final tags = [] - Map remote = checkForUpdates ? git.lsRemote().callAsMap() : null + Map remote = checkForUpdates ? bareGit.lsRemote().callAsMap() : null getBranchList() .findAll { it.name.startsWith('refs/heads/') || it.name.startsWith('refs/remotes/origin/') } .unique { shortenRefName(it.name) } .each { Ref it -> branches << refToMap(it,remote) } - remote = checkForUpdates ? git.lsRemote().setTags(true).callAsMap() : null + remote = checkForUpdates ? bareGit.lsRemote().setTags(true).callAsMap() : null getTagList() .findAll { it.name.startsWith('refs/tags/') } .each { Ref it -> tags << refToMap(it,remote) } - result.current = current // current branch name result.master = master // master branch name + result.pulled = listRevisionsToCompareInfo() // list of pulled revisions result.branches = branches // collection of branches result.tags = tags // collect of tags return result @@ -830,11 +1065,11 @@ class AssetManager { return human ? obj.name.substring(0,10) : obj.name } - protected String refToString(Ref ref, String current, String master, boolean tag, int level ) { + protected String refToString(Ref ref, String current, String master, List pulled, boolean tag, int level ) { def result = new StringBuilder() def name = shortenRefName(ref.name) - result << (name == current ? '*' : ' ') + result << (name in pulled ? 'P' : ' ') if( level ) { def peel = git.getRepository().peel(ref) @@ -880,14 +1115,14 @@ class AssetManager { @Deprecated List getUpdates(int level) { - def remote = git.lsRemote().callAsMap() + def remote = bareGit.lsRemote().callAsMap() List branches = getBranchList() .findAll { it.name.startsWith('refs/heads/') || it.name.startsWith('refs/remotes/origin/') } .unique { shortenRefName(it.name) } .findAll { Ref ref -> hasRemoteChange(ref,remote) } .collect { Ref ref -> formatUpdate(remote.get(ref.name),level) } - remote = git.lsRemote().setTags(true).callAsMap() + remote = bareGit.lsRemote().setTags(true).callAsMap() List tags = getTagList() .findAll { it.name.startsWith('refs/tags/') } .findAll { Ref ref -> hasRemoteChange(ref,remote) } @@ -899,39 +1134,7 @@ class AssetManager { return result } - /** - * Checkout a specific revision - * @param revision The revision to be checked out - */ - void checkout( String revision = null ) { - assert localPath - - def current = getCurrentRevision() - if( current != defaultBranch ) { - if( !revision ) { - throw new AbortOperationException("Project `$project` is currently stuck on revision: $current -- you need to explicitly specify a revision with the option `-r` in order to use it") - } - } - if( !revision || revision == current ) { - // nothing to do - return - } - - // verify that is clean - if( !isClean() ) - throw new AbortOperationException("Project `$project` contains uncommitted changes -- Cannot switch to revision: $revision") - - try { - git.checkout().setName(revision) .call() - } - catch( RefNotFoundException e ) { - checkoutRemoteBranch(revision) - } - - } - - - protected Ref checkoutRemoteBranch( String revision ) { + protected Ref checkoutRemoteBranch() { try { def fetch = git.fetch() @@ -997,12 +1200,12 @@ class AssetManager { if( provider.hasCredentials() ) update.setCredentialsProvider( provider.getGitCredentials() ) def updatedList = update.call() - log.debug "Update submodules $updatedList" + log.debug "Updating submodules $updatedList" } protected String getRemoteCommitId(RevisionInfo rev) { final tag = rev.type == RevisionInfo.Type.TAG - final cmd = git.lsRemote().setTags(tag) + final cmd = bareGit.lsRemote().setTags(tag) if( provider.hasCredentials() ) cmd.setCredentialsProvider( provider.getGitCredentials() ) final list = cmd.call() @@ -1042,7 +1245,7 @@ class AssetManager { } protected String getGitConfigRemoteUrl() { - if( !localPath ) { + if( !bareRepo.exists() ) { return null } @@ -1052,10 +1255,8 @@ class AssetManager { } final iniFile = new IniFile().load(gitConfig) - final branch = manifest.getDefaultBranch() - final remote = iniFile.getString("branch \"${branch}\"", "remote", "origin") - final url = iniFile.getString("remote \"${remote}\"", "url") - log.debug "Git config: $gitConfig; branch: $branch; remote: $remote; url: $url" + final url = iniFile.getString("remote \"origin\"", "url") + log.debug "Git config: $gitConfig; url: $url" return url } @@ -1081,14 +1282,13 @@ class AssetManager { protected String guessHubProviderFromGitConfig(boolean failFast=false) { - assert localPath // find the repository remote URL from the git project config file final domain = getGitConfigRemoteDomain() if( !domain && failFast ) { def message = (localGitConfig.exists() ? "Can't find git repository remote host -- Check config file at path: $localGitConfig" - : "Can't find git repository config file -- Repository may be corrupted: $localPath" ) + : "Can't find git repository config file -- Repository may be corrupted: $bareRepo" ) throw new AbortOperationException(message) } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy index 51a787d45c..1d0e7e55b6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy @@ -18,6 +18,7 @@ package nextflow.scm import groovy.util.logging.Slf4j +import static nextflow.Const.DEFAULT_BRANCH /** * Implements a repository provider for GitHub service * @@ -59,8 +60,8 @@ class GitlabRepositoryProvider extends RepositoryProvider { String getDefaultBranch() { def result = invokeAndParseResponse(getEndpointUrl()) ?. default_branch if( !result ) { - log.debug "Unable to fetch repo default branch. Using `master` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" - return 'master' + log.debug "Unable to fetch repo default branch. Using `${DEFAULT_BRANCH}` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" + return DEFAULT_BRANCH } return result } diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy index ab2be742bd..d4c355fc8f 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy @@ -45,8 +45,10 @@ class CmdInfoTest extends Specification { def setupSpec() { tempDir = Files.createTempDirectory('test') AssetManager.root = tempDir.toFile() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) // download the project manager.download() } @@ -70,7 +72,7 @@ class CmdInfoTest extends Specification { screen.contains(" local path : $tempDir/nextflow-io/hello" ) screen.contains(" main script : main.nf") screen.contains(" revisions : ") - screen.contains(" * master (default)") + screen.contains(" P master (default)") } def 'should print json info' () { @@ -90,8 +92,9 @@ class CmdInfoTest extends Specification { json.localPath == "$tempDir/nextflow-io/hello" json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == 'master' - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } json.revisions.tags.size()>1 @@ -116,8 +119,9 @@ class CmdInfoTest extends Specification { json.localPath == "$tempDir/nextflow-io/hello" json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == 'master' - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } json.revisions.tags.size()>1 diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy index de3fea6853..cca1e08a1e 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/CmdPullTest.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_SUBDIR + import nextflow.plugin.Plugins import spock.lang.IgnoreIf @@ -40,13 +42,13 @@ class CmdPullTest extends Specification { given: def accessToken = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def dir = Files.createTempDirectory('test') - def cmd = new CmdPull(args: ['nextflow-io/hello'], root: dir.toFile(), hubUser: accessToken) + def cmd = new CmdPull(args: ['nextflow-io/hello'], root: dir.toFile(), revision: '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8', hubUser: accessToken) when: cmd.run() then: - dir.resolve('nextflow-io/hello/.git').exists() - dir.resolve('nextflow-io/hello/README.md').exists() + dir.resolve('nextflow-io/hello/' + REVISION_SUBDIR + '/' + '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8' + '/.git').exists() + dir.resolve('nextflow-io/hello/' + REVISION_SUBDIR + '/' + '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8' + '/README.md').exists() cleanup: dir?.deleteDir() diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy index aa8bf1467c..bc62712b5b 100644 --- a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy @@ -16,6 +16,10 @@ package nextflow.scm +import static nextflow.scm.AssetManager.BARE_REPO +import static nextflow.scm.AssetManager.REVISION_MAP +import static nextflow.scm.AssetManager.REVISION_SUBDIR + import spock.lang.IgnoreIf import nextflow.exception.AbortOperationException @@ -103,6 +107,75 @@ class AssetManagerTest extends Specification { } + def testListRevisions() { + given: + String revisionMap1 = '''branch1,12345\nbranch2,67890''' + String revisionMap2 = '''branchA,abcde\nbranchB,fghij''' + + def folder = tempDir.getRoot() + folder.resolve('cbcrg/pipe1/.nextflow/').mkdirs() + folder.resolve('cbcrg/pipe2/.nextflow/').mkdirs() + folder.resolve('cbcrg/pipe1/' + REVISION_MAP).text = revisionMap1 + folder.resolve('cbcrg/pipe2/' + REVISION_MAP).text = revisionMap2 + + when: + def manager = new AssetManager('cbcrg/pipe1') + def list = manager.listRevisions() + then: + list == ['branch1','branch2'] + + when: + manager = new AssetManager('cbcrg/pipe2') + list = manager.listRevisions() + then: + list == ['branchA', 'branchB'] + } + + def testListRevisionsAndCommits() { + given: + String revisionMap1 = '''branch1,12345\nbranch2,67890''' + String revisionMap2 = '''branchA,abcde\nbranchB,fghij''' + + def folder = tempDir.getRoot() + folder.resolve('cbcrg/pipe1/.nextflow/').mkdirs() + folder.resolve('cbcrg/pipe2/.nextflow/').mkdirs() + folder.resolve('cbcrg/pipe1/' + REVISION_MAP).text = revisionMap1 + folder.resolve('cbcrg/pipe2/' + REVISION_MAP).text = revisionMap2 + + when: + def manager = new AssetManager('cbcrg/pipe1') + def dict = manager.listRevisionsAndCommits() + then: + dict == Map.of('branch1','12345','branch2','67890') + + when: + manager = new AssetManager('cbcrg/pipe2') + dict = manager.listRevisionsAndCommits() + then: + dict == Map.of('branchA','abcde','branchB','fghij') + } + + def testListCommits() { + given: + def folder = tempDir.getRoot() + folder.resolve('cbcrg/pipe1/' + REVISION_SUBDIR + '/12345').mkdirs() + folder.resolve('cbcrg/pipe1/' + REVISION_SUBDIR + '/67890').mkdirs() + folder.resolve('cbcrg/pipe2/' + REVISION_SUBDIR + '/abcde').mkdirs() + folder.resolve('cbcrg/pipe2/' + REVISION_SUBDIR + '/fghij').mkdirs() + + when: + def manager = new AssetManager('cbcrg/pipe1') + def list = manager.listCommits() + then: + list.sort() == ['12345','67890'] + + when: + manager = new AssetManager('cbcrg/pipe2') + list = manager.listCommits() + then: + list.sort() == ['abcde','fghij'] + } + def testResolveName() { @@ -157,110 +230,166 @@ class AssetManagerTest extends Specification { } + def testUpdateRevisionMap() { + + given: + def folder = tempDir.getRoot() + String revision = null + def manager = new AssetManager().build('nextflow-io/hello') + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) + String revisionMap1 = '''v1.2,1b420d060d3fad67027154ac48e3bdea06f058da\n''' + + when: + manager.updateRevisionMap('v1.2','1b420d060d3fad67027154ac48e3bdea06f058da') + then: + folder.resolve('nextflow-io/hello/' + REVISION_MAP).exists() + folder.resolve('nextflow-io/hello/' + REVISION_MAP).text == revisionMap1 + } + + + def testRevisionToCommitWithMap() { + + given: + def folder = tempDir.getRoot() + String revision = null + def manager = new AssetManager().build('nextflow-io/hello') + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) + String revisionMap1 = '''v1.2,1b420d060d3fad67027154ac48e3bdea06f058da\n''' + + when: + folder.resolve('nextflow-io/hello/.nextflow').mkdirs() + folder.resolve('nextflow-io/hello/' + REVISION_MAP).text = revisionMap1 + then: + manager.revisionToCommitWithMap('v1.2') == '1b420d060d3fad67027154ac48e3bdea06f058da' + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPull() { + def testCloneBareRepo() { given: def folder = tempDir.getRoot() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) when: - manager.download() + manager.checkBareRepo() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello/' + BARE_REPO).isDirectory() + folder.resolve('nextflow-io/hello/' + BARE_REPO + '/config').exists() + } + + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) + def testRevisionToCommitWithBareRepo() { + + given: + def folder = tempDir.getRoot() + String revision = null + def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) when: - manager.download() + manager.checkBareRepo() then: - noExceptionThrown() - + manager.revisionToCommitWithBareRepo('v1.2') == '1b420d060d3fad67027154ac48e3bdea06f058da' } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullTagTwice() { + def testPull() { given: def folder = tempDir.getRoot() + String revision = '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8' // easier to fix commit for a generic test def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) when: - manager.download("v1.2") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello/' + REVISION_SUBDIR + '/' + '7588c46ffefb4e3c06d4ab32c745c4d5e56cdad8' + '/.git').isDirectory() when: - manager.download("v1.2") + def result = manager.download() then: noExceptionThrown() + result == "Already-up-to-date" } - // The hashes used here are NOT associated with tags. + @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullHashTwice() { + def testPullTagTwice() { given: def folder = tempDir.getRoot() + String revision = 'v1.2' def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) + // tag v1.2 -> commit 1b420d060d3fad67027154ac48e3bdea06f058da when: - manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello/' + REVISION_SUBDIR + '/' + '1b420d060d3fad67027154ac48e3bdea06f058da' + '/.git').isDirectory() when: - def result = manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + manager.download() then: noExceptionThrown() - result == "Already-up-to-date" } - - // Downloading a branch first and then pulling the branch - // should work fine, unlike with tags. + // The hashes used here are NOT associated with tags. @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullBranchTwice() { + def testPullHashTwice() { given: def folder = tempDir.getRoot() + String revision = '6b9515aba6c7efc6a9b3f273ce116fc0c224bf68' def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) when: - manager.download("mybranch") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello/' + REVISION_SUBDIR + '/' + '6b9515aba6c7efc6a9b3f273ce116fc0c224bf68' + '/.git').isDirectory() when: - manager.download("mybranch") + def result = manager.download() then: noExceptionThrown() + result == "Already-up-to-date" } - // First clone a repo with a tag, then forget to include the -r argument - // when you execute nextflow. - // Note that while the download will work, execution will fail subsequently - // at a separate check - this just tests that we don't fail because of a detached head. + + // Downloading a branch first and then pulling the branch + // should work fine, unlike with tags. @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullTagThenBranch() { + def testPullBranchTwice() { given: def folder = tempDir.getRoot() + String revision = 'mybranch' def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) + // as of Jun 2024, branch "mybranch" -> commit "1c3e9e7404127514d69369cd87f8036830f5cf64" when: - manager.download("v1.2") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello/' + REVISION_SUBDIR + '/' + '1c3e9e7404127514d69369cd87f8036830f5cf64' + '/.git').isDirectory() when: - manager.download() + def result = manager.download() then: noExceptionThrown() + result == "Already-up-to-date" } @@ -269,8 +398,10 @@ class AssetManagerTest extends Specification { given: def dir = tempDir.getRoot() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/hello', [providers:[github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) when: manager.clone(dir.toFile()) @@ -381,14 +512,15 @@ class AssetManagerTest extends Specification { } ''' def dir = tempDir.getRoot() - dir.resolve('foo/bar').mkdirs() - dir.resolve('foo/bar/nextflow.config').text = config - dir.resolve('foo/bar/.git').mkdir() - dir.resolve('foo/bar/.git/config').text = GIT_CONFIG_TEXT + dir.resolve('foo/bar/' + REVISION_SUBDIR + '/' + 'mockup_dir').mkdirs() + dir.resolve('foo/bar/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/nextflow.config').text = config + dir.resolve('foo/bar/' + BARE_REPO).mkdirs() + dir.resolve('foo/bar/' + BARE_REPO + '/config').text = GIT_CONFIG_TEXT when: def holder = new AssetManager() holder.build('foo/bar') + holder.setLocalPath(new File(dir.toString() + '/foo/bar/' + REVISION_SUBDIR + '/' + 'mockup_dir')) then: holder.getMainScriptName() == 'hello.nf' holder.manifest.getDefaultBranch() == 'super-stuff' @@ -404,12 +536,11 @@ class AssetManagerTest extends Specification { def dir = tempDir.getRoot() dir.resolve('foo/bar').mkdirs() dir.resolve('foo/bar/nextflow.config').text = 'empty: 1' - dir.resolve('foo/bar/.git').mkdir() - dir.resolve('foo/bar/.git/config').text = GIT_CONFIG_TEXT when: def holder = new AssetManager() holder.build('foo/bar') + holder.setLocalPath(new File(dir.toString() + '/foo/bar')) then: holder.getMainScriptName() == 'main.nf' @@ -423,11 +554,11 @@ class AssetManagerTest extends Specification { given: def dir = tempDir.root - dir.resolve('.git').mkdir() - dir.resolve('.git/config').text = GIT_CONFIG_LONG + dir.resolve('foo/bar/' + BARE_REPO).mkdirs() + dir.resolve('foo/bar/' + BARE_REPO + '/config').text = GIT_CONFIG_LONG when: - def manager = new AssetManager().setLocalPath(dir.toFile()) + def manager = new AssetManager().build('foo/bar') then: manager.getGitConfigRemoteUrl() == 'git@github.com:nextflow-io/nextflow.git' @@ -437,11 +568,11 @@ class AssetManagerTest extends Specification { given: def dir = tempDir.root - dir.resolve('.git').mkdir() - dir.resolve('.git/config').text = GIT_CONFIG_LONG + dir.resolve('foo/bar/' + BARE_REPO).mkdirs() + dir.resolve('foo/bar/' + BARE_REPO + '/config').text = GIT_CONFIG_LONG when: - def manager = new AssetManager().setLocalPath(dir.toFile()) + def manager = new AssetManager().build('foo/bar') then: manager.getGitConfigRemoteDomain() == 'github.com' @@ -462,9 +593,6 @@ class AssetManagerTest extends Specification { def commit = repo.commit().setSign(false).setAll(true).setMessage('First commit').call() repo.close() - // append fake remote data - dir.resolve('.git/config').text = GIT_CONFIG_TEXT - when: def p = Mock(RepositoryProvider) { getRepositoryUrl() >> 'https://github.com/nextflow-io/nextflow' } and: @@ -550,20 +678,18 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() + String revision = 'dev' def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) - - when: - manager.download("dev") - then: - folder.resolve('nextflow-io/nf-test-branch/.git').isDirectory() - and: - folder.resolve('nextflow-io/nf-test-branch/workflow.nf').text == "println 'Hello'\n" + manager.setRevisionAndLocalPath('nextflow-io/nf-test-branch', revision) + // as of June 2024, branch "dev" -> commit "6f882561d589365c3950d170df8445e3c0dc8028" when: manager.download() then: - noExceptionThrown() + folder.resolve('nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/' + '6f882561d589365c3950d170df8445e3c0dc8028' + '/.git').isDirectory() + and: + folder.resolve('nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/' + '6f882561d589365c3950d170df8445e3c0dc8028' + '/workflow.nf').text == "println 'Hello'\n" } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) @@ -571,13 +697,14 @@ class AssetManagerTest extends Specification { given: def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') + String revision = 'dev' def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) expect: - manager.checkValidRemoteRepo('dev') + manager.checkValidRemoteRepo() and: manager.getMainScriptName() == 'workflow.nf' - } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) @@ -585,20 +712,18 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() + String revision = 'v0.1' def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) - - when: - manager.download("v0.1") - then: - folder.resolve('nextflow-io/nf-test-branch/.git').isDirectory() - and: - folder.resolve('nextflow-io/nf-test-branch/workflow.nf').text == "println 'Hello'\n" + manager.setRevisionAndLocalPath('nextflow-io/hello', revision) + // tag "v0.1" -> commit "6f882561d589365c3950d170df8445e3c0dc8028" when: manager.download() then: - noExceptionThrown() + folder.resolve('nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/' + '6f882561d589365c3950d170df8445e3c0dc8028' + '/.git').isDirectory() + and: + folder.resolve('nextflow-io/nf-test-branch/' + REVISION_SUBDIR + '/' + '6f882561d589365c3950d170df8445e3c0dc8028' + '/workflow.nf').text == "println 'Hello'\n" } } diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy index d3e82abf72..4e1f7e9e49 100644 --- a/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/scm/UpdateModuleTest.groovy @@ -16,6 +16,8 @@ package nextflow.scm +import static nextflow.scm.AssetManager.REVISION_SUBDIR + import java.nio.file.Files import java.nio.file.Path @@ -103,21 +105,22 @@ class UpdateModuleTest extends Specification { when: def manager = new AssetManager("file:${baseFolder}/pipe_x") - manager.download() + manager.setLocalPath(new File(target.toString() + '/local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir')) + manager.download(null, true) manager.updateModules() then: target.resolve('local/pipe_x').exists() - target.resolve('local/pipe_x/.git').exists() - target.resolve('local/pipe_x/main.nf').exists() + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/.git').exists() + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/main.nf').exists() - target.resolve('local/pipe_x/prj_aaa').exists() - target.resolve('local/pipe_x/prj_aaa/file1.txt').text == 'Hello' - target.resolve('local/pipe_x/prj_aaa/file2.log').text == 'World' + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_aaa').exists() + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_aaa/file1.txt').text == 'Hello' + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_aaa/file2.log').text == 'World' - target.resolve('local/pipe_x/prj_bbb').exists() - target.resolve('local/pipe_x/prj_bbb/file1.txt').text == 'Ciao' - target.resolve('local/pipe_x/prj_bbb/file2.log').text == 'Mondo' + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_bbb').exists() + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_bbb/file1.txt').text == 'Ciao' + target.resolve('local/pipe_x/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_bbb/file2.log').text == 'Mondo' } @@ -141,16 +144,17 @@ class UpdateModuleTest extends Specification { when: def manager = new AssetManager( "file:${baseFolder}/pipe_2" ) - manager.download() + manager.setLocalPath(new File(target.toString() + '/local/pipe_2/' + REVISION_SUBDIR + '/' + 'mockup_dir')) + manager.download(null, true) manager.updateModules() then: target.resolve('local/pipe_2').exists() - target.resolve('local/pipe_2/.git').exists() - target.resolve('local/pipe_2/main.nf').exists() + target.resolve('local/pipe_2/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/.git').exists() + target.resolve('local/pipe_2/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/main.nf').exists() - target.resolve('local/pipe_2/prj_aaa').list().size()==0 - target.resolve('local/pipe_2/prj_bbb').list().size()==0 + target.resolve('local/pipe_2/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_aaa').list().size()==0 + target.resolve('local/pipe_2/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_bbb').list().size()==0 } def 'should clone selected submodules' () { @@ -174,21 +178,22 @@ class UpdateModuleTest extends Specification { when: def manager = new AssetManager( "file:${baseFolder}/pipe_3" ) - manager.download() + manager.setLocalPath(new File(target.toString() + '/local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir')) + manager.download(null, true) manager.updateModules() then: target.resolve('local/pipe_3').exists() - target.resolve('local/pipe_3/.git').exists() - target.resolve('local/pipe_3/main.nf').exists() + target.resolve('local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/.git').exists() + target.resolve('local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/main.nf').exists() - target.resolve('local/pipe_3/prj_aaa').list().size()==0 + target.resolve('local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_aaa').list().size()==0 - target.resolve('local/pipe_3/prj_bbb').exists() - target.resolve('local/pipe_3/prj_bbb/file1.txt').text == 'Ciao' + target.resolve('local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_bbb').exists() + target.resolve('local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_bbb/file1.txt').text == 'Ciao' - target.resolve('local/pipe_3/prj_ccc').exists() - target.resolve('local/pipe_3/prj_ccc/file-x.txt').text == 'x' + target.resolve('local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_ccc').exists() + target.resolve('local/pipe_3/' + REVISION_SUBDIR + '/' + 'mockup_dir' + '/prj_ccc/file-x.txt').text == 'x' }