From f8c4bcbcccd18d7207a73a81b45fb473c8283130 Mon Sep 17 00:00:00 2001 From: Marcel Ribeiro-Dantas Date: Tue, 18 Feb 2025 23:57:43 -0300 Subject: [PATCH 1/7] Fix capitalization of GitHub and Gitpod --- .github/workflows/docker.yml | 4 ++-- docs/envsetup/01_setup.md | 2 +- docs/envsetup/index.md | 4 ++-- docs/hello_nextflow/00_orientation.pt.md | 2 +- docs/nextflow_run/03_run_nf-core.md | 4 ++-- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index bc91d3ae7..1a02cb913 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -1,8 +1,8 @@ -# Build and push the Docker image for GitPod +# Build and push the Docker image for Gitpod # - Only pushes if push or release # - Builds without push for PRs to check the Dockerfile -name: Build GitPod Docker image +name: Build Gitpod Docker image on: pull_request: diff --git a/docs/envsetup/01_setup.md b/docs/envsetup/01_setup.md index 7b2db2034..87387818c 100644 --- a/docs/envsetup/01_setup.md +++ b/docs/envsetup/01_setup.md @@ -24,7 +24,7 @@ Using a machine with more cores allows you to take greater advantage of Nextflow **For our Hello Nextflow and Nextflow For Science training courses, we recommend using a 4-core machine.** -The free Github plan includes 120 core-hours of Codespaces compute per month, which amounts to 30 hours of a 4-core machine. +The free GitHub plan includes 120 core-hours of Codespaces compute per month, which amounts to 30 hours of a 4-core machine. (See below for more information about quotas.) !!! warning diff --git a/docs/envsetup/index.md b/docs/envsetup/index.md index c04f6dda9..958d660c8 100644 --- a/docs/envsetup/index.md +++ b/docs/envsetup/index.md @@ -13,11 +13,11 @@ Let's get started! For more detailed instructions for GitHub Codespaces, see the [GitHub Codespaces env-setup docs](01_setup.md). If you cannot use GitHub Codespaces and wish to use a local development environment, see the [documentation for local installation](02_local.md). -!!! info "Deprecation of GitPod" +!!! info "Deprecation of Gitpod" Nextflow Training used to use [Gitpod](https://gitpod.io) until February 2025. However, the makers of Gitpod have decided to retire the free functionality in favor of their new [Gitpod Flex](https://www.gitpod.io/blog/introducing-gitpod-flex) system. For that reason, we have switched to using GitHub Codespaces, which also offer a one-click developer environment with no prior setup. - Depending on when you signed up to GitPod and when exactly they retire the service, you may still be able to launch the training in their old cloud IDE, though we cannot guarantee reliable access going forward: + Depending on when you signed up to Gitpod and when exactly they retire the service, you may still be able to launch the training in their old cloud IDE, though we cannot guarantee reliable access going forward: [Open in Gitpod](https://gitpod.io/#https://github.com/nextflow-io/training). diff --git a/docs/hello_nextflow/00_orientation.pt.md b/docs/hello_nextflow/00_orientation.pt.md index 6c8343f7d..a697e11be 100644 --- a/docs/hello_nextflow/00_orientation.pt.md +++ b/docs/hello_nextflow/00_orientation.pt.md @@ -1,6 +1,6 @@ # Orientação -O ambiente Github Codespaces contém todo o software, código e dados necessários para este curso. Você não precisa instalar nada por conta própria. No entanto, é necessária uma conta (gratuita) para logar - e recomendamos que você reserve alguns minutos para se familiarizar com a interface. +O ambiente GitHub Codespaces contém todo o software, código e dados necessários para este curso. Você não precisa instalar nada por conta própria. No entanto, é necessária uma conta (gratuita) para logar - e recomendamos que você reserve alguns minutos para se familiarizar com a interface. Caso ainda não tenha feito isso, siga [este link](../../envsetup/) antes de prosseguir. diff --git a/docs/nextflow_run/03_run_nf-core.md b/docs/nextflow_run/03_run_nf-core.md index dee38cb4d..0b78d3b73 100644 --- a/docs/nextflow_run/03_run_nf-core.md +++ b/docs/nextflow_run/03_run_nf-core.md @@ -305,9 +305,9 @@ Template features can be flexibly included or excluded at the time of creation, 5. On the Template features screen, turn **off**: - `Use a GitHub repository` - - `Add Github CI tests` + - `Add GitHub CI tests` - `Use reference genomes` - - `Add Github badges` + - `Add GitHub badges` - `Include citations` - `Include a gitpod environment` - `Include GitHub Codespaces` From 753ceddfd267a49cf80efb2a6ec3e89514eca09b Mon Sep 17 00:00:00 2001 From: Marcel Ribeiro-Dantas Date: Wed, 19 Feb 2025 00:46:21 -0300 Subject: [PATCH 2/7] Minor fixes to Part 3 --- docs/hello_nextflow/03_hello_workflow.md | 34 ++++++++++++------------ 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/hello_nextflow/03_hello_workflow.md b/docs/hello_nextflow/03_hello_workflow.md index 71d80829d..26e654a3f 100644 --- a/docs/hello_nextflow/03_hello_workflow.md +++ b/docs/hello_nextflow/03_hello_workflow.md @@ -62,13 +62,13 @@ If that worked for you, you're ready to learn how to assemble a multi-step workf We're going to add a step to convert the greeting to uppercase. To that end, we need to do three things: -- Define the command we'lre going to use to do the uppercase conversion. +- Define the command we're going to use to do the uppercase conversion. - Write a new process that wraps the uppercasing command. -- Add the new process to the workflow and set it up to take the output of the `sayHello()` process as input. +- Call the new process in the workflow block and set it up to take the output of the `sayHello()` process as input. ### 1.1. Define the uppercasing command and test it in the terminal -To do the conversion of the greetings to uppercase, we're going to a classic UNIX tool called `tr` for 'text replacement', with the following syntax: +To do the conversion of the greetings to uppercase, we're going to use a classic UNIX tool called `tr` for 'text replacement', with the following syntax: ```bash title="Syntax" tr '[a-z]' '[A-Z]' @@ -338,7 +338,7 @@ Nextflow doesn't mind, so it doesn't matter. This is where things could get a little tricky, because we need to be able to handle an arbitrary number of input files. Specifically, we can't write the command up front, so we need to tell Nextflow how to compose it at runtime based on what inputs flow into the process. -In other words, if we have an input channel containing the item `[file1.txt, file2.txt, file3.txt]`, we need Nextflow to turn that into `cat file1.txt file2.txt file3.txt`. +In other words, if we have an input channel containing the element `[file1.txt, file2.txt, file3.txt]`, we need Nextflow to turn that into `cat file1.txt file2.txt file3.txt`. Fortunately, Nextflow is quite happy to do that for us if we simply write `cat ${input_files}` in the script command. @@ -368,7 +368,7 @@ In theory this should handle any arbitrary number of input files. Some command-line tools require providing an argument (like `-input`) for each input file. In that case, we would have to do a little bit of extra work to compose the command. - You can see an example of this in the 'Nextflow for Genomics' training course. + You can see an example of this in the [Nextflow for Genomics](https://training.nextflow.io/latest/nf4_science/genomics/) training course. @@ -427,13 +427,13 @@ We were only expecting one, but there are three. And have a look at the contents of the final output file too: -```console title="COLLECTED-output.txt" +```console title="results/COLLECTED-output.txt" Holà ``` Oh no. The collection step was run individually on each greeting, which is NOT what we wanted. -We need to do something to tell Nextflow explicitly that we want that third step to run on all the items in the channel output by `convertToUpper()`. +We need to do something to tell Nextflow explicitly that we want that third step to run on all the elements in the channel output by `convertToUpper()`. ### 2.3. Use an operator to collect the greetings into a single input @@ -521,22 +521,22 @@ This time the third step was only called once! Looking at the output of the `view()` statements, we see the following: - Three `Before collect:` statements, one for each greeting: at that point the file paths are individual items in the channel. -- A single `After collect:` statement: the three file paths are now packaged into a single item. +- A single `After collect:` statement: the three file paths are now packaged into a singl element. Have a look at the contents of the final output file too: -```console title="COLLECTED-output.txt" +```console title="results/COLLECTED-output.txt" BONJOUR HELLO HOLà ``` -This time we have all three greetings in the final output file. Success! +This time we have all three greetings in the final output file. Success! Remove the optional view calls to make the next outputs less verbose. !!! note If you run this several times without `-resume`, you will see that the order of the greetings changes from one run to the next. - This shows you that the order in which items flow through the pipeline is not guaranteed to be consistent. + This shows you that the order in which elements flow through process calls is not guaranteed to be consistent, unless you make use of fair threading (check the [fair](https://www.nextflow.io/docs/latest/reference/process.html#fair) directive). ### Takeaway @@ -742,14 +742,14 @@ Conveniently, Nextflow lets us add arbitrary code in the `script:` block of the That means we can use the built-in `size()` function to get the number of files in the `input_files` array. -In the process block, make the following code change: +In the `collectGreetings` process block, make the following code change: _Before:_ ```groovy title="hello-workflow.nf" linenums="55" script: """ - cat ${input_files} > 'COLLECTED-${batch_id}-output.txt' + cat ${input_files} > 'COLLECTED-${batch_name}-output.txt' """ ``` @@ -759,7 +759,7 @@ _After:_ script: count_greetings = input_files.size() """ - cat ${input_files} > 'COLLECTED-${batch_id}-output.txt' + cat ${input_files} > 'COLLECTED-${batch_name}-output.txt' """ ``` @@ -777,14 +777,14 @@ _Before:_ ```groovy title="hello-workflow.nf" linenums="52" output: - path "COLLECTED-${batch_id}-output.txt" + path "COLLECTED-${batch_name}-output.txt" ``` _After:_ ```groovy title="hello-workflow.nf" linenums="52" output: - path "COLLECTED-${batch_id}-output.txt" , emit: outfile + path "COLLECTED-${batch_name}-output.txt" , emit: outfile val count_greetings , emit: count ``` @@ -793,7 +793,7 @@ But as the saying goes, why not both? ### 4.2. Report the output at the end of the workflow -Now that we have two outputs coming out of the `collectGreetings` process, the `collectGreetings.out` output channel contains two 'tracks': +Now that we have two outputs coming out of the `collectGreetings` process, the `collectGreetings.out` output multi-channel contains two channels: - `collectGreetings.out.outfile` contains the final output file - `collectGreetings.out.count` contains the count of greetings From 05b8a44405b75562786f9942529623164f6e5d1e Mon Sep 17 00:00:00 2001 From: Marcel Ribeiro-Dantas Date: Wed, 19 Feb 2025 01:13:23 -0300 Subject: [PATCH 3/7] Tiny typo fix in Part 3 --- docs/hello_nextflow/03_hello_workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hello_nextflow/03_hello_workflow.md b/docs/hello_nextflow/03_hello_workflow.md index 26e654a3f..e4499a906 100644 --- a/docs/hello_nextflow/03_hello_workflow.md +++ b/docs/hello_nextflow/03_hello_workflow.md @@ -521,7 +521,7 @@ This time the third step was only called once! Looking at the output of the `view()` statements, we see the following: - Three `Before collect:` statements, one for each greeting: at that point the file paths are individual items in the channel. -- A single `After collect:` statement: the three file paths are now packaged into a singl element. +- A single `After collect:` statement: the three file paths are now packaged into a single element. Have a look at the contents of the final output file too: From 528b5c249242cec5d3850cd307fb2ec4841a74f2 Mon Sep 17 00:00:00 2001 From: Marcel Ribeiro-Dantas Date: Wed, 19 Feb 2025 01:37:11 -0300 Subject: [PATCH 4/7] Minor fixes to Part 5 --- docs/hello_nextflow/05_hello_containers.md | 80 ++++++++++++++-------- 1 file changed, 50 insertions(+), 30 deletions(-) diff --git a/docs/hello_nextflow/05_hello_containers.md b/docs/hello_nextflow/05_hello_containers.md index ba6887bba..92fbfe1c7 100644 --- a/docs/hello_nextflow/05_hello_containers.md +++ b/docs/hello_nextflow/05_hello_containers.md @@ -86,7 +86,7 @@ The `''` part is the URI address of the container image. As an example, let's pull a container image that contains [cowpy](https://github.com/jeffbuttars/cowpy), a python implementation of a tool called `cowsay` that generates ASCII art to display arbitrary text inputs in a fun way. There are various repositories where you can find published containers. -We used the [Seqera Containers](https://seqera.io/containers/) service to generate this Docker container from the `cowpy` Conda package: `'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'`. +We used the [Seqera Containers](https://seqera.io/containers/) service to generate this Docker container image from the `cowpy` Conda package: `'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'`. Run the complete pull command: @@ -163,7 +163,7 @@ You can also run a container interactively, which gives you a shell prompt insid #### 1.3.1. Spin up the container -To run interactively, we just add `-it` to the `docker pull` command. +To run interactively, we just add `-it` to the `docker run` command. Optionally, we can specify the shell we want to use inside the container by appending _e.g._ `/bin/bash` to the command. ```bash @@ -200,7 +200,7 @@ For example, the tool documentation says we can change the character ('cowacter' cowpy "Hello Containers" -c tux ``` -Now the output shows the Linux penguin, Tux, instead of the default cow, because we specified `-c tux` parameter. +Now the output shows the Linux penguin, Tux, instead of the default cow, because we specified the `-c tux` parameter. ```console title="Output" __________________ @@ -250,7 +250,7 @@ One way to do this is to **mount** a **volume** from the host system into the co -v : ``` -In our case `` will be the current working directory, so we can just use a dot (`.`), and `` is just a name we make up; let's call it `/data`. +In our case `` will be the current working directory, so we can just use a dot (`.`), and `` is just a name we make up; let's call it `/data`. To mount a volume, we replace the paths and add the volume mounting argument to the docker run command as follows: @@ -265,6 +265,7 @@ You can check that it works by listing the contents of `/data`: ```bash ls /data ``` +Depending on what part of this training you've done before, the output below my look slightly different. ```console title="Output" demo-params.json hello-channels.nf hello-workflow.nf modules results @@ -290,11 +291,11 @@ cat /data/greetings.csv | cowpy -c turkey This produces the desired ASCII art of a turkey rattling off our example greetings: ```console title="Output" - _________ -/ HOLà \ -| HELLO | -\ BONJOUR / - --------- + _________ +/ Hello \ +| Bonjour | +\ Holà / + --------- \ ,+*^^*+___+++_ \ ,*^^^^ ) \ _+* ^**+_ @@ -502,6 +503,21 @@ ERROR ~ Error executing process > 'cowpy' Caused by: Process `cowpy` terminated with an error exit status (127) + +Command executed: + + cat COLLECTED-test-batch-output.txt | cowpy -c "turkey" > cowpy-COLLECTED-test-batch-output.txt + +Command exit status: + 127 + +Command output: + (empty) + +Command error: + .command.sh: line 2: cowpy: command not found + +(trimmed output) ``` This error code, `error exit status (127)` means the executable we asked for was not found. @@ -530,7 +546,6 @@ _After:_ process cowpy { publishDir 'containers/results', mode: 'copy' - container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273' ``` @@ -590,27 +605,32 @@ There were 3 greetings in this batch You can find the cowpy'ed output in the `results` directory. ```console title="results/cowpy-COLLECTED-test-batch-output.txt" - _______ - / \ + _________ +/ HOLà \ | HELLO | -| HOLà | -| BONJOUR | - \ / - ======= - \ - \ - \ - \ - ,. - (_|,. - ,' /, )_______ _ - __j o``-' `.'-)' - (") \' - `-j | - `-._( / - |_\ |--^. / - /_]'|_| /_)_/ - /_]' /_]' +\ BONJOUR / + --------- + \ ,+*^^*+___+++_ + \ ,*^^^^ ) + \ _+* ^**+_ + \ +^ _ _++*+_+++_, ) + _+^^*+_ ( ,+*^ ^ \+_ ) + { ) ( ,( ,_+--+--, ^) ^\ + { (\@) } f ,( ,+-^ __*_*_ ^^\_ ^\ ) + {:;-/ (_+*-+^^^^^+*+*<_ _++_)_ ) ) / + ( / ( ( ,___ ^*+_+* ) < < \ + U _/ ) *--< ) ^\-----++__) ) ) ) + ( ) _(^)^^)) ) )\^^^^^))^*+/ / / + ( / (_))_^)) ) ) ))^^^^^))^^^)__/ +^^ + ( ,/ (^))^)) ) ) ))^^^^^^^))^^) _) + *+__+* (_))^) ) ) ))^^^^^^))^^^^^)____*^ + \ \_)^)_)) ))^^^^^^^^^^))^^^^) + (_ ^\__^^^^^^^^^^^^))^^^^^^^) + ^\___ ^\__^^^^^^))^^^^^^^^)\\ + ^^^^^\uuu/^^\uuu/^^^^\^\^\^\^\^\^\^\ + ___) >____) >___ ^\_\_\_\_\_\_\) + ^^^//\\_^^//\\_^ ^(\_\_\_\) + ^^^ ^^ ^^^ ^ ``` You see that the character is saying all the greetings, just as it did when we ran the `cowpy` command on the `greetings.csv` file from inside the container. From 766fa1654946f935a02147a8a501ab14a562fb9e Mon Sep 17 00:00:00 2001 From: Marcel Ribeiro-Dantas Date: Wed, 19 Feb 2025 02:18:45 -0300 Subject: [PATCH 5/7] Fix linting errors --- docs/hello_nextflow/05_hello_containers.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/hello_nextflow/05_hello_containers.md b/docs/hello_nextflow/05_hello_containers.md index 92fbfe1c7..bd7d7c116 100644 --- a/docs/hello_nextflow/05_hello_containers.md +++ b/docs/hello_nextflow/05_hello_containers.md @@ -265,6 +265,7 @@ You can check that it works by listing the contents of `/data`: ```bash ls /data ``` + Depending on what part of this training you've done before, the output below my look slightly different. ```console title="Output" @@ -291,11 +292,11 @@ cat /data/greetings.csv | cowpy -c turkey This produces the desired ASCII art of a turkey rattling off our example greetings: ```console title="Output" - _________ + _________ / Hello \ | Bonjour | \ Holà / - --------- + --------- \ ,+*^^*+___+++_ \ ,*^^^^ ) \ _+* ^**+_ @@ -605,11 +606,11 @@ There were 3 greetings in this batch You can find the cowpy'ed output in the `results` directory. ```console title="results/cowpy-COLLECTED-test-batch-output.txt" - _________ + _________ / HOLà \ | HELLO | \ BONJOUR / - --------- + --------- \ ,+*^^*+___+++_ \ ,*^^^^ ) \ _+* ^**+_ From ce0d18f7e07fd0df0f007ff7503cb1750311825a Mon Sep 17 00:00:00 2001 From: Marcel Ribeiro-Dantas Date: Wed, 19 Feb 2025 08:51:11 -0300 Subject: [PATCH 6/7] Replace absolute link with relative link --- docs/hello_nextflow/03_hello_workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hello_nextflow/03_hello_workflow.md b/docs/hello_nextflow/03_hello_workflow.md index e4499a906..1041e2f76 100644 --- a/docs/hello_nextflow/03_hello_workflow.md +++ b/docs/hello_nextflow/03_hello_workflow.md @@ -368,7 +368,7 @@ In theory this should handle any arbitrary number of input files. Some command-line tools require providing an argument (like `-input`) for each input file. In that case, we would have to do a little bit of extra work to compose the command. - You can see an example of this in the [Nextflow for Genomics](https://training.nextflow.io/latest/nf4_science/genomics/) training course. + You can see an example of this in the [Nextflow for Genomics](../nf4_science/genomics/) training course. From e1eea987580bd8e78fed2fa8c6da31cb854628e8 Mon Sep 17 00:00:00 2001 From: Marcel Ribeiro-Dantas Date: Wed, 19 Feb 2025 08:54:25 -0300 Subject: [PATCH 7/7] Add suggestions from Geraldine and Phil - Remove mention of fair threading - Remove mention of multi-channel - Convert absolute to relative link --- docs/hello_nextflow/03_hello_workflow.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hello_nextflow/03_hello_workflow.md b/docs/hello_nextflow/03_hello_workflow.md index 1041e2f76..95ed94c26 100644 --- a/docs/hello_nextflow/03_hello_workflow.md +++ b/docs/hello_nextflow/03_hello_workflow.md @@ -368,7 +368,7 @@ In theory this should handle any arbitrary number of input files. Some command-line tools require providing an argument (like `-input`) for each input file. In that case, we would have to do a little bit of extra work to compose the command. - You can see an example of this in the [Nextflow for Genomics](../nf4_science/genomics/) training course. + You can see an example of this in the [Nextflow for Genomics](../../nf4_science/genomics/) training course. @@ -536,7 +536,7 @@ This time we have all three greetings in the final output file. Success! Remove !!! note If you run this several times without `-resume`, you will see that the order of the greetings changes from one run to the next. - This shows you that the order in which elements flow through process calls is not guaranteed to be consistent, unless you make use of fair threading (check the [fair](https://www.nextflow.io/docs/latest/reference/process.html#fair) directive). + This shows you that the order in which elements flow through process calls is not guaranteed to be consistent. ### Takeaway @@ -793,7 +793,7 @@ But as the saying goes, why not both? ### 4.2. Report the output at the end of the workflow -Now that we have two outputs coming out of the `collectGreetings` process, the `collectGreetings.out` output multi-channel contains two channels: +Now that we have two outputs coming out of the `collectGreetings` process, the `collectGreetings.out` output contains two channels: - `collectGreetings.out.outfile` contains the final output file - `collectGreetings.out.count` contains the count of greetings