diff --git a/docs/cli.md b/docs/cli.md index df93409d7d..e06f15c3f8 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -366,7 +366,7 @@ The `clone` command downloads a pipeline from a Git-hosting platform into the *c : Service hub where the project is hosted. Options: `gitlab` or `bitbucket`. `-r` (`master`) -: Revision to clone - It can be a git branch, tag, or revision number. +: Revision of the project to clone (either a git branch, tag or commit SHA number). `-user` : Private repository user name. @@ -415,6 +415,9 @@ The `config` command is used for printing the project's configuration i.e. the ` `-properties` : Print config using Java properties notation. +`-r, -revision` +: Revision of the project (either a git branch, tag or commit SHA number). + `-a, -show-profiles` : Show all configuration profiles. @@ -535,12 +538,18 @@ The `drop` command is used to remove the projects which have been downloaded int **Options** +`-a, -all-revisions` +: For specified project, drop all revisions. + `-f` : Delete the repository without taking care of local changes. `-h, -help` : Print the command usage. +`-r, -revision` +: Revision of the project to drop (either a git branch, tag or commit SHA number). + **Examples** Drop the `nextflow-io/hello` project. @@ -664,7 +673,7 @@ $ nextflow info [options] [project] **Description** -The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project. +The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project; in this case, note how revisions marked as `P` are pulled locally. If no run name or session id is provided, it will clean the latest run. @@ -878,6 +887,9 @@ The `list` commands prints a list of the projects which are already downloaded i **Options** +`-a, -all-revisions` +: For each project, also list revisions. + `-h, -help` : Print the command usage. @@ -1072,7 +1084,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo : Service hub where the project is hosted. Options: `gitlab` or `bitbucket` `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Revision of the project to pull (either a git branch, tag or commit SHA number). : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-user` @@ -1217,7 +1229,7 @@ The `run` command is used to execute a local pipeline script or remote pipeline : Execute the script using the cached results, useful to continue executions that was stopped by an error. `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Revision of the project to run (either a git branch, tag or commit SHA number). : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-stub-run, -stub` @@ -1426,6 +1438,9 @@ The `view` command is used to inspect the pipelines that are already stored in t `-q` : Hide header line. +`-r, -revision` +: Revision of the project (either a git branch, tag or commit SHA number). + **Examples** Viewing the contents of a downloaded pipeline. diff --git a/docs/developer/diagrams/nextflow.scm.mmd b/docs/developer/diagrams/nextflow.scm.mmd index 1c52075372..234fe70a69 100644 --- a/docs/developer/diagrams/nextflow.scm.mmd +++ b/docs/developer/diagrams/nextflow.scm.mmd @@ -8,9 +8,12 @@ classDiagram class AssetManager { project : String + revision : String + commitId : String localPath : File + localBarePath : File mainScript : String - repositoryProvider : RepositoryProvider + provider : RepositoryProvider hub : String providerConfigs : List~ProviderConfig~ } diff --git a/docs/sharing.md b/docs/sharing.md index 2db5b9b527..4575b73f14 100644 --- a/docs/sharing.md +++ b/docs/sharing.md @@ -58,6 +58,11 @@ nextflow run nextflow-io/hello -r v1.1 It will execute two different project revisions corresponding to the Git tag/branch having that names. +:::{versionadded} 24.XX.0-edge +::: + +Nextflow downloads and locally maintains each explicitly requested Git branch, tag or commit ID in a separate directory path, thus enabling to run multiple revisions of the same pipeline at the same time. Each downloaded revision is stored in a sub-directory path to the default pipeline path one, named after the corresponding commit ID, according to the schema `/.nextflow/commits/`. + ## Commands to manage projects The following commands allows you to perform some basic operations that can be used to manage your projects. @@ -94,13 +99,13 @@ repository : http://github.com/nextflow-io/hello local path : $HOME/.nextflow/assets/nextflow-io/hello main script : main.nf revisions : -* master (default) +P master (default) mybranch - v1.1 [t] +P v1.1 [t] v1.2 [t] ``` -Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local folder where the project has been downloaded; 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the current checked-out revision is marked with a `*` on the left. +Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local path where the default project can be found (alternate revisions are in sister paths with an extra suffix `:`); 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the locally pulled revisions are marked with a `P` on the left. ### Pulling or updating a project diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy index 9c582aaaf6..b7be19d0b6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy @@ -15,6 +15,7 @@ */ package nextflow.cli + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -37,7 +38,7 @@ class CmdClone extends CmdBase implements HubOptions { @Parameter(required=true, description = 'name of the project to clone') List args - @Parameter(names='-r', description = 'Revision to clone - It can be a git branch, tag or revision number') + @Parameter(names='-r', description = 'Revision of the project to clone (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') @@ -52,11 +53,11 @@ class CmdClone extends CmdBase implements HubOptions { Plugins.init() // the pipeline name String pipeline = args[0] - final manager = new AssetManager(pipeline, this) + final manager = new AssetManager(pipeline, revision, this) // the target directory is the second parameter // otherwise default the current pipeline name - def target = new File(args.size()> 1 ? args[1] : manager.getBaseName()) + def target = new File(args.size()> 1 ? args[1] : manager.getBaseNameWithRevision()) if( target.exists() ) { if( target.isFile() ) throw new AbortOperationException("A file with the same name already exists: $target") @@ -68,9 +69,9 @@ class CmdClone extends CmdBase implements HubOptions { } manager.checkValidRemoteRepo() - print "Cloning ${manager.project}${revision ? ':'+revision:''} ..." - manager.clone(target, revision, deep) + print "Cloning ${manager.getProjectWithRevision()} ..." + manager.clone(target, deep) print "\r" - println "${manager.project} cloned to: $target" + println "${manager.getProjectWithRevision()} cloned to: $target" } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy index 5ff2156b18..851261c67b 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy @@ -44,6 +44,9 @@ class CmdConfig extends CmdBase { @Parameter(description = 'project name') List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names=['-a','-show-profiles'], description = 'Show all configuration profiles') boolean showAllProfiles @@ -183,7 +186,7 @@ class CmdConfig extends CmdBase { return file.parent ?: Paths.get('/') } - final manager = new AssetManager(path) + final manager = new AssetManager(path, revision) manager.isLocal() ? manager.localPath.toPath() : manager.configFile?.parent } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy index 9d67190a54..fdad006d55 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -39,6 +41,12 @@ class CmdDrop extends CmdBase { @Parameter(required=true, description = 'name of the project to drop') List args + @Parameter(names=['-r','-revision'], description = 'Revision of the project to drop (either a git branch, tag or commit SHA number)') + String revision + + @Parameter(names=['-a','-all-revisions'], description = 'For specified project, drop all revisions') + Boolean allrevisions + @Parameter(names='-f', description = 'Delete the repository without taking care of local changes') boolean force @@ -48,18 +56,34 @@ class CmdDrop extends CmdBase { @Override void run() { Plugins.init() - def manager = new AssetManager(args[0]) - if( !manager.localPath.exists() ) { - throw new AbortOperationException("No match found for: ${args[0]}") + + List dropList = [] + if ( allrevisions ) { + def referenceManager = new AssetManager(args[0]) + referenceManager.listRevisions().each { + dropList << new AssetManager(it.tokenize(REVISION_DELIM)[0], it.tokenize(REVISION_DELIM)[1]) + } + } else { + dropList << new AssetManager(args[0], revision) } - if( this.force || manager.isClean() ) { - manager.close() - if( !manager.localPath.deleteDir() ) - throw new AbortOperationException("Unable to delete project `${manager.project}` -- Check access permissions for path: ${manager.localPath}") - return + if ( !dropList ) { + throw new AbortOperationException("No revisions found for specified project: ${args[0]}") } - throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + dropList.each { manager -> + if( !manager.localPath.exists() ) { + throw new AbortOperationException("No match found for: ${manager.getProjectWithRevision()}") + } + + if( this.force || manager.isClean() ) { + manager.close() + if( !manager.localPath.deleteDir() ) + throw new AbortOperationException("Unable to delete project `${manager.getProjectWithRevision()}` -- Check access permissions for path: ${manager.localPath}") + return + } + + throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + } } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy index c8c1c31bc0..428c914f9a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + import java.lang.management.ManagementFactory import java.nio.file.spi.FileSystemProvider @@ -75,9 +77,16 @@ class CmdInfo extends CmdBase { } Plugins.init() - final manager = new AssetManager(args[0]) - if( !manager.isLocal() ) - throw new AbortOperationException("Unknown project `${args[0]}`") + def manager = new AssetManager(args[0], null) + if( !manager.isLocal() ) { + // if default branch not found locally, use first one from list of local pulls + if ( manager.listRevisions() ) { + manager = new AssetManager(args[0], manager.getPulledRevisions()[0]) + } + else { + throw new AbortOperationException("Unknown project `${args[0]}`") + } + } if( !format || format == 'text' ) { printText(manager,level) @@ -101,7 +110,7 @@ class CmdInfo extends CmdBase { out.println " project name: ${manager.project}" out.println " repository : ${manager.repositoryUrl}" - out.println " local path : ${manager.localPath}" + out.println " local path : ${manager.localPath.toString().tokenize(REVISION_DELIM)[0]}" out.println " main script : ${manager.mainScriptName}" if( manager.homePage && manager.homePage != manager.repositoryUrl ) out.println " home page : ${manager.homePage}" @@ -138,7 +147,7 @@ class CmdInfo extends CmdBase { def result = [:] result.projectName = manager.project result.repository = manager.repositoryUrl - result.localPath = manager.localPath?.toString() + result.localPath = manager.localPath?.toString().tokenize(REVISION_DELIM)[0] result.manifest = manager.manifest.toMap() result.revisions = manager.getBranchesAndTags(checkForUpdates) return result diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy index 5b08be249a..f6c851931a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy @@ -16,6 +16,9 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + +import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic import groovy.util.logging.Slf4j @@ -33,6 +36,9 @@ class CmdList extends CmdBase { static final public NAME = 'list' + @Parameter(names=['-a','-all-revisions'], description = 'For each project, also list revisions') + Boolean revisions + @Override final String getName() { NAME } @@ -45,7 +51,15 @@ class CmdList extends CmdBase { return } - all.each { println it } + if (revisions) { + all.collect{ it.tokenize(REVISION_DELIM) } + .groupBy{ it[0] } + .each{ println ' ' + it.value[0][0] ; it.value.each{ y -> println ( y.size()==1 ? ' (default)' : ' ' + y[1] ) } } + } else { + all.collect{ it.replaceAll( /$REVISION_DELIM.*/, '' ) } + .unique() + .each{ println ' ' + it } + } } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy index 9d2fa9a01d..f6310f3972 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy @@ -15,6 +15,9 @@ */ package nextflow.cli + +import static nextflow.scm.AssetManager.REVISION_DELIM + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -40,7 +43,7 @@ class CmdPull extends CmdBase implements HubOptions { @Parameter(names='-all', description = 'Update all downloaded projects', arity = 0) boolean all - @Parameter(names=['-r','-revision'], description = 'Revision of the project to run (either a git branch, tag or commit SHA number)') + @Parameter(names=['-r','-revision'], description = 'Revision of the project to pull (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') @@ -73,10 +76,11 @@ class CmdPull extends CmdBase implements HubOptions { Plugins.init() list.each { - log.info "Checking $it ..." - def manager = new AssetManager(it, this) + log.info "Checking $it${revision ? REVISION_DELIM + revision : ''} ..." + def manager = new AssetManager(it, revision, this) - def result = manager.download(revision,deep) + manager.updateLocalBareRepo() + def result = manager.download(deep) manager.updateModules() def scriptFile = manager.getScriptFile() diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy index e5d18bdb87..667d08ad99 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy @@ -17,6 +17,7 @@ package nextflow.cli import static org.fusesource.jansi.Ansi.* +import static nextflow.scm.AssetManager.REVISION_DELIM import java.nio.file.NoSuchFileException import java.nio.file.Path @@ -569,22 +570,22 @@ class CmdRun extends CmdBase implements HubOptions { /* * try to look for a pipeline in the repository */ - def manager = new AssetManager(pipelineName, this) + def manager = new AssetManager(pipelineName, revision, this) def repo = manager.getProject() boolean checkForUpdate = true if( !manager.isRunnable() || latest ) { if( offline ) - throw new AbortOperationException("Unknown project `$repo` -- NOTE: automatic download from remote repositories is disabled") - log.info "Pulling $repo ..." - def result = manager.download(revision,deep) + throw new AbortOperationException("Unknown project `$repo${revision ? REVISION_DELIM + revision : ''}` -- NOTE: automatic download from remote repositories is disabled") + log.info "Pulling $repo${revision ? REVISION_DELIM + revision : ''} ..." + manager.updateLocalBareRepo() + def result = manager.download(deep) if( result ) log.info " $result" checkForUpdate = false } - // checkout requested revision + // post download operations try { - manager.checkout(revision) manager.updateModules() final scriptFile = manager.getScriptFile(mainScript) if( checkForUpdate && !offline ) @@ -596,7 +597,7 @@ class CmdRun extends CmdBase implements HubOptions { throw e } catch( Exception e ) { - throw new AbortOperationException("Unknown error accessing project `$repo` -- Repository may be corrupted: ${manager.localPath}", e) + throw new AbortOperationException("Unknown error accessing project `$repo${revision ? REVISION_DELIM + revision : ''}` -- Repository may be corrupted: ${manager.localPath}", e) } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy index dcdcfcae67..165be1f9b3 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -42,6 +44,9 @@ class CmdView extends CmdBase { @Parameter(description = 'project name', required = true) List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names = '-q', description = 'Hide header line', arity = 0) boolean quiet @@ -51,9 +56,9 @@ class CmdView extends CmdBase { @Override void run() { Plugins.init() - def manager = new AssetManager(args[0]) + def manager = new AssetManager(args[0], revision) if( !manager.isLocal() ) - throw new AbortOperationException("Unknown project name `${args[0]}`") + throw new AbortOperationException("Unknown project `${args[0]}${revision ? REVISION_DELIM + revision : ''}`") if( all ) { if( !quiet ) diff --git a/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy b/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy index c50cef627d..4a2e5a83a8 100644 --- a/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy @@ -273,7 +273,7 @@ class K8sDriverLauncher { if( !interactive && !pipelineName.startsWith('/') && !cmd.remoteProfile && !cmd.runRemoteConfig ) { // -- check and parse project remote config Plugins.init() - final pipelineConfig = new AssetManager(pipelineName, cmd) .getConfigFile() + final pipelineConfig = new AssetManager(pipelineName, cmd.revision, cmd) .getConfigFile() builder.setUserConfigFiles(pipelineConfig) } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy index dd304ad59e..531f9562c6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy @@ -62,6 +62,11 @@ class AssetManager { @PackageScope static File root = DEFAULT_ROOT + @PackageScope + static final String subdirCommits = '.nextflow/commits' + + static public final String REVISION_DELIM = ':' + /** * The pipeline name. It must be in the form {@code username/repo} where 'username' * is a valid user name or organisation account, while 'repo' is the repository name @@ -69,11 +74,32 @@ class AssetManager { */ private String project + /** + * The name of the commit/branch/tag as requested via command line + * This is now a first class attribute of a pipeline + */ + private String revision + + /** + * The commit ID as de-referenced from the requested revision + */ + private String commitId + /** * Directory where the pipeline is cloned (i.e. downloaded) + * + * Schema: $NXF_ASSETS///.nextflow/commits/ */ private File localPath + /** + * Directory where the bare repository of the pipeline + * is cloned (i.e. downloaded) + * + * Schema: $NXF_ASSETS// + */ + private File localBarePath + private Git _git private String mainScript @@ -94,48 +120,77 @@ class AssetManager { /** * Create a new asset manager with the specified pipeline name * - * @param pipeline The pipeline to be managed by this manager e.g. {@code nextflow-io/hello} + * @param pipelineName The pipeline to be managed by this manager e.g. {@code nextflow-io/hello} + * @param revision Revision ID for the selected pipeline (git branch, tag or commit SHA number) */ - AssetManager( String pipelineName, HubOptions cliOpts = null) { + AssetManager( String pipelineName, String revision = null, HubOptions cliOpts = null) { assert pipelineName // read the default config file (if available) def config = ProviderConfig.getDefault() // build the object - build(pipelineName, config, cliOpts) + build(pipelineName, revision, config, cliOpts) } - AssetManager( String pipelineName, Map config ) { + AssetManager( String pipelineName, String revision, Map config) { assert pipelineName // build the object - build(pipelineName, config) + build(pipelineName, revision, config) } /** * Build the asset manager internal data structure * * @param pipelineName A project name or a project repository Git URL + * @param revision Revision ID for the selected pipeline (git branch, tag or commit SHA number) * @param config A {@link Map} holding the configuration properties defined in the {@link ProviderConfig#DEFAULT_SCM_FILE} file * @param cliOpts User credentials provided on the command line. See {@link HubOptions} trait * @return The {@link AssetManager} object itself */ @PackageScope - AssetManager build( String pipelineName, Map config = null, HubOptions cliOpts = null ) { + AssetManager build( String pipelineName, String revision = null, Map config = null, HubOptions cliOpts = null ) { this.providerConfigs = ProviderConfig.createFromMap(config) - this.project = resolveName(pipelineName) - this.localPath = checkProjectDir(project) + this.revision = revision + this.project = resolveName(pipelineName, this.revision) + + this.localBarePath = resolveLocalBarePath(project) this.hub = checkHubProvider(cliOpts) this.provider = createHubProvider(hub) + this.provider.setRevision(this.revision) + checkLocalBarePath() + setupCredentials(cliOpts) - validateProjectDir() + validateBareProjectDir() + + /* TODO MARCO : Outstanding bits right now: + 1. updating of bare ideally would be at rev/tag level, however does everything by default + -> need to test this: git.fetch() + .setRefSpecs("refs/heads/:refs/heads/") + 2. the wrong revision/commit is printed at run time, as in "Launching ..." + - also, no notice when remote branch is updated (probably related) + -> how is RevisionInfo used in the run algorithm? + ==> should probably make sure that requested revision is printed in pipeline run + 3. brittle algorithm: + - instantiation: decode revision to commit ID (so as to define localPath) + - if pull/run: update local bare -> can result in mismatch of revision vs commit + + 4. update mechanics of list/info commands + + END. refactor with original AssetManager (which has no revision arg). also unit tests. + */ + + // note: this will call updateLocalBareRepo() if revision cannot be resolved in first instance + this.commitId = commitFromRevisionUsingBareLocal(this.revision) + this.localPath = checkProjectDir(project, this.commitId) return this } + @PackageScope File getLocalGitConfig() { - localPath ? new File(localPath,'.git/config') : null + localBarePath ? new File(localBarePath,'config') : null } @PackageScope AssetManager setProject(String name) { @@ -173,37 +228,93 @@ class AssetManager { * and return the directory where the project is stored locally * * @param projectName A project name matching the pattern {@code owner/project} + * @param revision Revision ID for the selected pipeline (git branch, tag or commit SHA number) * @return The project dir {@link File} */ @PackageScope - File checkProjectDir(String projectName) { + File checkProjectDir(String projectName, String commit) { if( !isValidProjectName(projectName)) { throw new IllegalArgumentException("Not a valid project name: $projectName") } + new File(root, project + '/' + subdirCommits + '/' + commit) + } + + @PackageScope + File resolveLocalBarePath(String projectName) { new File(root, project) } + @PackageScope + void checkLocalBarePath() { + /* + * if the bare repository of the pipeline does not exists locally pull it from the remote repo + */ + if( !localBarePath.exists() ) { + localBarePath.parentFile.mkdirs() + // make sure it contains a valid repository + checkValidRemoteRepoBare() + + final cloneBareURL = getGitRepositoryUrlBare() + log.debug "Pulling bare repo for $project -- Using remote clone url: ${cloneBareURL}" + + // clone it + def bare = Git.cloneRepository() + if( provider.hasCredentials() ) + bare.setCredentialsProvider( provider.getGitCredentials() ) + + bare + .setBare( true ) + .setURI(cloneBareURL) + .setGitDir(localBarePath) + .call() + } + } + + void updateLocalBareRepo() { + log.debug "Fetching (updating) bare repo for $project" + + Git.open(localBarePath) + .fetch() + .call() + } + + @PackageScope + String commitFromRevisionUsingBareLocal(String revision) { + String bareRevision = revision ?: Constants.HEAD + def bare = Git.open(localBarePath) + def rev = bare.getRepository().resolve(bareRevision) + if (rev == null) { + updateLocalBareRepo() + rev = bare.getRepository().resolve(bareRevision) + if (rev == null) + throw new AbortOperationException("Cannot resolve revision: $bareRevision") + } + def commit = rev.getName() + bare.close() + return commit + } + /** * Verifies that the project hub provider eventually specified by the user using the {@code -hub} command * line option or implicitly by entering a repository URL, matches with clone URL of a project already cloned (downloaded). */ @PackageScope - void validateProjectDir() { + void validateBareProjectDir() { - if( !localPath.exists() ) { + if( !localBarePath.exists() ) { return } // if project dir exists it must contain the Git config file final configProvider = guessHubProviderFromGitConfig(true) if( !configProvider ) - throw new IllegalStateException("Cannot find a provider config for repository at path: $localPath") + throw new IllegalStateException("Cannot find a provider config for repository at path: $localBarePath") // checks that the selected hub matches with the one defined in the git config file if( hub != configProvider ) { - throw new AbortOperationException("A project with name: `$localPath` has already been downloaded from a different provider: `$configProvider`") + throw new AbortOperationException("A project with name: `$localBarePath` has already been downloaded from a different provider: `$configProvider`") } } @@ -243,7 +354,7 @@ class AssetManager { * @return The fully qualified project name e.g. {@code cbcrg/foo} */ @PackageScope - String resolveName( String name ) { + String resolveName( String name, String revision = null ) { assert name // @@ -285,7 +396,7 @@ class AssetManager { name = parts[0] } - def qualifiedName = find(name) + def qualifiedName = find(name, revision) if( !qualifiedName ) { return "$DEFAULT_ORGANIZATION/$name".toString() } @@ -300,6 +411,10 @@ class AssetManager { String getProject() { project } + String getRevision() { revision } + + String getProjectWithRevision() { project + ( revision ? REVISION_DELIM + revision : '' ) } + String getHub() { hub } @PackageScope @@ -363,28 +478,39 @@ class AssetManager { return this } - AssetManager checkValidRemoteRepo(String revision=null) { - // Configure the git provider to use the required revision as source for all needed remote resources: - // - config if present in repo (nextflow.config by default) - // - main script (main.nf by default) - provider.revision = revision + AssetManager checkValidRemoteRepo() { + // Check that the remote git provider contains the main script file (main.nf by default) final scriptName = getMainScriptName() provider.validateFor(scriptName) return this } + AssetManager checkValidRemoteRepoBare() { + // Check that the remote git provider contains the main script file (main.nf by default) + final scriptName = getMainScriptNameBare() + provider.validateFor(scriptName) + return this + } + @Memoized String getGitRepositoryUrl() { - if( localPath.exists() ) { + if( ( localPath != null ? localPath.exists() : false ) ) { return localPath.toURI().toString() } provider.getCloneUrl() } + @Memoized + String getGitRepositoryUrlBare() { + provider.getCloneUrl() + } + File getLocalPath() { localPath } + File getLocalBarePath() { localBarePath } + ScriptFile getScriptFile(String scriptName=null) { def result = new ScriptFile(getMainScriptFile(scriptName)) @@ -413,6 +539,10 @@ class AssetManager { return mainScript ?: getManifest().getMainScript() } + String getMainScriptNameBare() { + return mainScript ?: getManifest0().getMainScript() + } + String getHomePage() { getManifest().getHomePage() ?: provider.getRepositoryUrl() } @@ -434,7 +564,7 @@ class AssetManager { String text = null ConfigObject result = null try { - text = localPath.exists() ? new File(localPath, MANIFEST_FILE_NAME).text : provider.readText(MANIFEST_FILE_NAME) + text = ( localPath != null ? localPath.exists() : false ) ? new File(localPath, MANIFEST_FILE_NAME).text : provider.readText(MANIFEST_FILE_NAME) } catch( FileNotFoundException e ) { log.debug "Project manifest does not exist: ${e.message}" @@ -459,7 +589,7 @@ class AssetManager { } Path getConfigFile() { - if( localPath.exists() ) { + if( ( localPath != null ? localPath.exists() : false ) ) { return new File(localPath, MANIFEST_FILE_NAME).toPath() } else { @@ -478,10 +608,11 @@ class AssetManager { } - String getBaseName() { + String getBaseNameWithRevision() { def result = project.tokenize('/') if( result.size() > 2 ) throw new IllegalArgumentException("Not a valid project name: $project") - return result.size()==1 ? result[0] : result[1] + result = ( result.size()==1 ? result[0] : result[1] ) + return result + ( revision ? REVISION_DELIM + revision : '' ) } boolean isLocal() { @@ -538,16 +669,48 @@ class AssetManager { return result } - static protected def find( String name ) { + /** + * @return The list of available revisions for a given project name + */ + List listRevisions( String projectName = project ) { + log.debug "Listing revisions for project: $projectName" + + def result = new LinkedList() + if( !root.exists() ) + return result + + list().each { + if( it.tokenize(REVISION_DELIM)[0] == projectName ) { + result << it + } + } + + return result + } + + // Updated for new localPath schema (see localPath declaration at top of this class file) + static protected def find( String name, String revision = null ) { def exact = [] def partial = [] list().each { def items = it.split('/') - if( items[1] == name ) - exact << it - else if( items[1].startsWith(name ) ) - partial << it + /** + * itemsRev[0] is the name of each list'ed project + * itemsRev[1] is the revision + */ + def itemsRev = items[1].tokenize(REVISION_DELIM) + // Check on matching revision: either null or same revision string + if( (!revision && !itemsRev[1]) || (revision && itemsRev[1] == revision) ) { + // Exact name match + if( itemsRev[0] == name ) + // Return item without revision + exact << it.tokenize(REVISION_DELIM)[0] + // Partial name match + else if( itemsRev[0].startsWith(name ) ) + // Return item without revision + partial << it.tokenize(REVISION_DELIM)[0] + } } def list = exact ?: partial @@ -565,22 +728,18 @@ class AssetManager { /** * Download a pipeline from a remote Github repository * - * @param revision The revision to download * @result A message representing the operation result */ - String download(String revision=null, Integer deep=null) { + String download(Integer deep=null) { assert project /* - * if the pipeline already exists locally pull it from the remote repo + * if the pipeline does not exists locally pull it from the remote repo */ if( !localPath.exists() ) { localPath.parentFile.mkdirs() - // make sure it contains a valid repository - checkValidRemoteRepo(revision) - final cloneURL = getGitRepositoryUrl() - log.debug "Pulling $project -- Using remote clone url: ${cloneURL}" + log.debug "Pulling $project -- Using local bare repo" // clone it def clone = Git.cloneRepository() @@ -588,87 +747,41 @@ class AssetManager { clone.setCredentialsProvider( provider.getGitCredentials() ) clone - .setURI(cloneURL) + .setURI(localBarePath.toString()) .setDirectory(localPath) .setCloneSubmodules(manifest.recurseSubmodules) if( deep ) clone.setDepth(deep) clone.call() - if( revision ) { - // use an explicit checkout command *after* the clone instead of cloning a specific branch - // because the clone command does not allow the use of SHA commit id (only branch and tag names) - try { git.checkout() .setName(revision) .call() } - catch ( RefNotFoundException e ) { checkoutRemoteBranch(revision) } - } + // use an explicit checkout command *after* the clone instead of cloning a specific branch + // because the clone command does not allow the use of SHA commit id (only branch and tag names) + git.checkout() .setName(commitId) .call() + //try { git.checkout() .setName(commitId) .call() } + //catch ( RefNotFoundException e ) { checkoutRemoteBranch() } // return status message - return "downloaded from ${cloneURL}" - } - - log.debug "Pull pipeline $project -- Using local path: $localPath" - - // verify that is clean - if( !isClean() ) - throw new AbortOperationException("$project contains uncommitted changes -- cannot pull from repository") - - if( revision && revision != getCurrentRevision() ) { - /* - * check out a revision before the pull operation - */ - try { - git.checkout() .setName(revision) .call() - } - /* - * If the specified revision does not exist - * Try to checkout it from a remote branch and return - */ - catch ( RefNotFoundException e ) { - final ref = checkoutRemoteBranch(revision) - final commitId = ref?.getObjectId() - return commitId - ? "checked out at ${commitId.name()}" - : "checked out revision ${revision}" - } - } - - def pull = git.pull() - def revInfo = getCurrentRevisionAndName() - - if ( revInfo.type == RevisionInfo.Type.COMMIT ) { - log.debug("Repo appears to be checked out to a commit hash, but not a TAG, so we will assume the repo is already up to date and NOT pull it!") - return MergeResult.MergeStatus.ALREADY_UP_TO_DATE.toString() + return "downloaded from local bare repo" } - if ( revInfo.type == RevisionInfo.Type.TAG ) { - pull.setRemoteBranchName( "refs/tags/" + revInfo.name ) - } - - if( provider.hasCredentials() ) - pull.setCredentialsProvider( provider.getGitCredentials() ) - - if( manifest.recurseSubmodules ) { - pull.setRecurseSubmodules(FetchRecurseSubmodulesMode.YES) - } - def result = pull.call() - if(!result.isSuccessful()) - throw new AbortOperationException("Cannot pull project `$project` -- ${result.toString()}") + // verify that is runnable + if( !isRunnable() ) + throw new AbortOperationException("$project is not runnable and not empty -- cannot pull from repository") - return result?.mergeResult?.mergeStatus?.toString() + return "already available" } /** * Clone a pipeline from a remote pipeline repository to the specified folder * * @param directory The folder when the pipeline will be cloned - * @param revision The revision to be cloned. It can be a branch, tag, or git revision number */ - void clone(File directory, String revision = null, Integer deep=null) { + void clone(File directory, Integer deep=null) { def clone = Git.cloneRepository() def uri = getGitRepositoryUrl() - log.debug "Clone project `$project` -- Using remote URI: ${uri} into: $directory" + log.debug "Cloning `${project}` from ${uri} into ${directory}" if( !uri ) throw new AbortOperationException("Cannot find the specified project: $project") @@ -706,6 +819,18 @@ class AssetManager { names.get( head.objectId ) ?: head.objectId.name() } + /** + * @return The names of all locally pulled revisions for a given project + * + * If revision is null, default is assumed + */ + List getPulledRevisions() { + return listRevisions().collect{ + it -> String y = it.tokenize(REVISION_DELIM)[1] + it = ( y != null ? y : getDefaultBranch() ) + } + } + RevisionInfo getCurrentRevisionAndName() { Ref head = git.getRepository().findRef(Constants.HEAD); if( !head ) @@ -733,30 +858,31 @@ class AssetManager { /** * @return A list of existing branches and tags names. For example * 
-     *     * master (default)
-     *       patch-x
-     *       v1.0 (t)
-     *       v1.1 (t)
+     *     * P master (default)
+     *         patch-x
+     *         v1.0 (t)
+     *         v1.1 (t)
      *
* - * The star character on the left highlight the current revision, the string {@code (default)} - * ticks that it is the default working branch (usually the master branch), while the string {@code (t)} - * shows that the revision is a git tag (instead of a branch) + * The character {@code P} on the left indicates the revision is pulled locally, + * the string {@code (default)} ticks that it is the default working branch, + * while the string {@code (t)} shows that the revision is a git tag (instead of a branch) */ @Deprecated List getRevisions(int level) { def current = getCurrentRevision() def master = getDefaultBranch() + def pulled = getPulledRevisions() List branches = getBranchList() .findAll { it.name.startsWith('refs/heads/') || it.name.startsWith('refs/remotes/origin/') } .unique { shortenRefName(it.name) } - .collect { Ref it -> refToString(it,current,master,false,level) } + .collect { Ref it -> refToString(it,current,master,pulled,false,level) } List tags = getTagList() .findAll { it.name.startsWith('refs/tags/') } - .collect { refToString(it,current,master,true,level) } + .collect { refToString(it,current,master,pulled,true,level) } def result = new ArrayList(branches.size() + tags.size()) result.addAll(branches) @@ -777,7 +903,6 @@ class AssetManager { Map getBranchesAndTags(boolean checkForUpdates) { final result = [:] - final current = getCurrentRevision() final master = getDefaultBranch() final branches = [] @@ -794,8 +919,8 @@ class AssetManager { .findAll { it.name.startsWith('refs/tags/') } .each { Ref it -> tags << refToMap(it,remote) } - result.current = current // current branch name result.master = master // master branch name + result.pulled = getPulledRevisions() // list of pulled revisions result.branches = branches // collection of branches result.tags = tags // collect of tags return result @@ -830,11 +955,11 @@ class AssetManager { return human ? obj.name.substring(0,10) : obj.name } - protected String refToString(Ref ref, String current, String master, boolean tag, int level ) { + protected String refToString(Ref ref, String current, String master, List pulled, boolean tag, int level ) { def result = new StringBuilder() def name = shortenRefName(ref.name) - result << (name == current ? '*' : ' ') + result << (name in pulled ? 'P' : ' ') if( level ) { def peel = git.getRepository().peel(ref) @@ -899,39 +1024,7 @@ class AssetManager { return result } - /** - * Checkout a specific revision - * @param revision The revision to be checked out - */ - void checkout( String revision = null ) { - assert localPath - - def current = getCurrentRevision() - if( current != defaultBranch ) { - if( !revision ) { - throw new AbortOperationException("Project `$project` is currently stuck on revision: $current -- you need to explicitly specify a revision with the option `-r` in order to use it") - } - } - if( !revision || revision == current ) { - // nothing to do - return - } - - // verify that is clean - if( !isClean() ) - throw new AbortOperationException("Project `$project` contains uncommitted changes -- Cannot switch to revision: $revision") - - try { - git.checkout().setName(revision) .call() - } - catch( RefNotFoundException e ) { - checkoutRemoteBranch(revision) - } - - } - - - protected Ref checkoutRemoteBranch( String revision ) { + protected Ref checkoutRemoteBranch() { try { def fetch = git.fetch() @@ -997,7 +1090,7 @@ class AssetManager { if( provider.hasCredentials() ) update.setCredentialsProvider( provider.getGitCredentials() ) def updatedList = update.call() - log.debug "Update submodules $updatedList" + log.debug "Updating submodules $updatedList" } protected String getRemoteCommitId(RevisionInfo rev) { @@ -1042,7 +1135,7 @@ class AssetManager { } protected String getGitConfigRemoteUrl() { - if( !localPath ) { + if( !localBarePath ) { return null } @@ -1052,10 +1145,8 @@ class AssetManager { } final iniFile = new IniFile().load(gitConfig) - final branch = manifest.getDefaultBranch() - final remote = iniFile.getString("branch \"${branch}\"", "remote", "origin") - final url = iniFile.getString("remote \"${remote}\"", "url") - log.debug "Git config: $gitConfig; branch: $branch; remote: $remote; url: $url" + final url = iniFile.getString("remote \"origin\"", "url") + log.debug "Git config: $gitConfig; url: $url" return url } @@ -1081,14 +1172,14 @@ class AssetManager { protected String guessHubProviderFromGitConfig(boolean failFast=false) { - assert localPath + assert localBarePath // find the repository remote URL from the git project config file final domain = getGitConfigRemoteDomain() if( !domain && failFast ) { def message = (localGitConfig.exists() ? "Can't find git repository remote host -- Check config file at path: $localGitConfig" - : "Can't find git repository config file -- Repository may be corrupted: $localPath" ) + : "Can't find git repository config file -- Repository may be corrupted: $localBarePath" ) throw new AbortOperationException(message) } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy index 51a787d45c..1d0e7e55b6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy @@ -18,6 +18,7 @@ package nextflow.scm import groovy.util.logging.Slf4j +import static nextflow.Const.DEFAULT_BRANCH /** * Implements a repository provider for GitHub service * @@ -59,8 +60,8 @@ class GitlabRepositoryProvider extends RepositoryProvider { String getDefaultBranch() { def result = invokeAndParseResponse(getEndpointUrl()) ?. default_branch if( !result ) { - log.debug "Unable to fetch repo default branch. Using `master` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" - return 'master' + log.debug "Unable to fetch repo default branch. Using `${DEFAULT_BRANCH}` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" + return DEFAULT_BRANCH } return result } diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy index ab2be742bd..59a12a847f 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy @@ -45,8 +45,9 @@ class CmdInfoTest extends Specification { def setupSpec() { tempDir = Files.createTempDirectory('test') AssetManager.root = tempDir.toFile() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', revision, [providers: [github: [auth: token]]]) // download the project manager.download() } @@ -70,7 +71,7 @@ class CmdInfoTest extends Specification { screen.contains(" local path : $tempDir/nextflow-io/hello" ) screen.contains(" main script : main.nf") screen.contains(" revisions : ") - screen.contains(" * master (default)") + screen.contains(" P master (default)") } def 'should print json info' () { @@ -90,8 +91,9 @@ class CmdInfoTest extends Specification { json.localPath == "$tempDir/nextflow-io/hello" json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == 'master' - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } json.revisions.tags.size()>1 @@ -116,8 +118,9 @@ class CmdInfoTest extends Specification { json.localPath == "$tempDir/nextflow-io/hello" json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == 'master' - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } json.revisions.tags.size()>1 diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy index aa8bf1467c..6531f84c21 100644 --- a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy @@ -103,6 +103,32 @@ class AssetManagerTest extends Specification { } + def testListRevisions() { + given: + def folder = tempDir.getRoot() + folder.resolve('cbcrg/pipe1').mkdirs() + folder.resolve('cbcrg/pipe2').mkdirs() + folder.resolve('cbcrg/pipe2:v2').mkdirs() + folder.resolve('cbcrg/pipe3:v3').mkdirs() + + def manager = new AssetManager() + + when: + def list = manager.listRevisions('cbcrg/pipe1') + then: + list == ['cbcrg/pipe1'] + + when: + list = manager.listRevisions('cbcrg/pipe3') + then: + list == ['cbcrg/pipe3:v3'] + + when: + list = manager.listRevisions('cbcrg/pipe2') + then: + list == ['cbcrg/pipe2', 'cbcrg/pipe2:v2'] + } + def testResolveName() { @@ -110,6 +136,7 @@ class AssetManagerTest extends Specification { def folder = tempDir.getRoot() folder.resolve('cbcrg/pipe1').mkdirs() folder.resolve('cbcrg/pipe2').mkdirs() + folder.resolve('cbcrg/pipe2:v2').mkdirs() folder.resolve('ncbi/blast').mkdirs() def manager = new AssetManager() @@ -119,6 +146,11 @@ class AssetManagerTest extends Specification { then: result == 'x/y' + when: + result = manager.resolveName('x/y', 'v2') + then: + result == 'x/y' + when: result = manager.resolveName('blast') then: @@ -144,6 +176,11 @@ class AssetManagerTest extends Specification { then: thrown(AbortOperationException) + when: + result = manager.resolveName('pipe2', 'v2') + then: + result == 'cbcrg/pipe2' + when: result = manager.resolveName('../blast/script.nf') then: @@ -162,8 +199,9 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', revision, [providers: [github: [auth: token]]]) when: manager.download() @@ -171,10 +209,10 @@ class AssetManagerTest extends Specification { folder.resolve('nextflow-io/hello/.git').isDirectory() when: - manager.download() + def result = manager.download() then: noExceptionThrown() - + result == "Already-up-to-date" } @@ -184,15 +222,15 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', "v1.2", [providers: [github: [auth: token]]]) when: - manager.download("v1.2") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello:v1.2/.git').isDirectory() when: - manager.download("v1.2") + manager.download() then: noExceptionThrown() } @@ -204,15 +242,15 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', "6b9515aba6c7efc6a9b3f273ce116fc0c224bf68", [providers: [github: [auth: token]]]) when: - manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello:6b9515aba6c7efc6a9b3f273ce116fc0c224bf68/.git').isDirectory() when: - def result = manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + def result = manager.download() then: noExceptionThrown() result == "Already-up-to-date" @@ -227,40 +265,18 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', "mybranch", [providers: [github: [auth: token]]]) when: - manager.download("mybranch") - then: - folder.resolve('nextflow-io/hello/.git').isDirectory() - - when: - manager.download("mybranch") - then: - noExceptionThrown() - } - - // First clone a repo with a tag, then forget to include the -r argument - // when you execute nextflow. - // Note that while the download will work, execution will fail subsequently - // at a separate check - this just tests that we don't fail because of a detached head. - @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullTagThenBranch() { - - given: - def folder = tempDir.getRoot() - def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) - - when: - manager.download("v1.2") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello:mybranch/.git').isDirectory() when: - manager.download() + def result = manager.download() then: noExceptionThrown() + result == "Already-up-to-date" } @@ -269,8 +285,9 @@ class AssetManagerTest extends Specification { given: def dir = tempDir.getRoot() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers:[github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', revision, [providers:[github: [auth: token]]]) when: manager.clone(dir.toFile()) @@ -551,19 +568,14 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) - - when: - manager.download("dev") - then: - folder.resolve('nextflow-io/nf-test-branch/.git').isDirectory() - and: - folder.resolve('nextflow-io/nf-test-branch/workflow.nf').text == "println 'Hello'\n" + def manager = new AssetManager().build('nextflow-io/nf-test-branch', "dev", [providers: [github: [auth: token]]]) when: manager.download() then: - noExceptionThrown() + folder.resolve('nextflow-io/nf-test-branch:dev/.git').isDirectory() + and: + folder.resolve('nextflow-io/nf-test-branch:dev/workflow.nf').text == "println 'Hello'\n" } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) @@ -571,13 +583,12 @@ class AssetManagerTest extends Specification { given: def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/nf-test-branch', 'dev', [providers: [github: [auth: token]]]) expect: - manager.checkValidRemoteRepo('dev') + manager.checkValidRemoteRepo() and: manager.getMainScriptName() == 'workflow.nf' - } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) @@ -586,19 +597,14 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) - - when: - manager.download("v0.1") - then: - folder.resolve('nextflow-io/nf-test-branch/.git').isDirectory() - and: - folder.resolve('nextflow-io/nf-test-branch/workflow.nf').text == "println 'Hello'\n" + def manager = new AssetManager().build('nextflow-io/nf-test-branch', "v0.1", [providers: [github: [auth: token]]]) when: manager.download() then: - noExceptionThrown() + folder.resolve('nextflow-io/nf-test-branch:v0.1/.git').isDirectory() + and: + folder.resolve('nextflow-io/nf-test-branch:v0.1/workflow.nf').text == "println 'Hello'\n" } }