Skip to content

Minor fixes to parts 3, 4, and 5 of Hello Nextflow #514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Feb 19, 2025
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Build and push the Docker image for GitPod
# Build and push the Docker image for Gitpod
# - Only pushes if push or release
# - Builds without push for PRs to check the Dockerfile

name: Build GitPod Docker image
name: Build Gitpod Docker image

on:
pull_request:
Expand Down
2 changes: 1 addition & 1 deletion docs/envsetup/01_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Using a machine with more cores allows you to take greater advantage of Nextflow

**For our Hello Nextflow and Nextflow For Science training courses, we recommend using a 4-core machine.**

The free Github plan includes 120 core-hours of Codespaces compute per month, which amounts to 30 hours of a 4-core machine.
The free GitHub plan includes 120 core-hours of Codespaces compute per month, which amounts to 30 hours of a 4-core machine.
(See below for more information about quotas.)

!!! warning
Expand Down
4 changes: 2 additions & 2 deletions docs/envsetup/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ Let's get started!
For more detailed instructions for GitHub Codespaces, see the [GitHub Codespaces env-setup docs](01_setup.md).
If you cannot use GitHub Codespaces and wish to use a local development environment, see the [documentation for local installation](02_local.md).

!!! info "Deprecation of GitPod"
!!! info "Deprecation of Gitpod"

Nextflow Training used to use [Gitpod](https://gitpod.io) until February 2025.
However, the makers of Gitpod have decided to retire the free functionality in favor of their new [Gitpod Flex](https://www.gitpod.io/blog/introducing-gitpod-flex) system.
For that reason, we have switched to using GitHub Codespaces, which also offer a one-click developer environment with no prior setup.

Depending on when you signed up to GitPod and when exactly they retire the service, you may still be able to launch the training in their old cloud IDE, though we cannot guarantee reliable access going forward:
Depending on when you signed up to Gitpod and when exactly they retire the service, you may still be able to launch the training in their old cloud IDE, though we cannot guarantee reliable access going forward:
[Open in Gitpod](https://gitpod.io/#https://github.com/nextflow-io/training).
2 changes: 1 addition & 1 deletion docs/hello_nextflow/00_orientation.pt.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Orientação

O ambiente Github Codespaces contém todo o software, código e dados necessários para este curso. Você não precisa instalar nada por conta própria. No entanto, é necessária uma conta (gratuita) para logar - e recomendamos que você reserve alguns minutos para se familiarizar com a interface.
O ambiente GitHub Codespaces contém todo o software, código e dados necessários para este curso. Você não precisa instalar nada por conta própria. No entanto, é necessária uma conta (gratuita) para logar - e recomendamos que você reserve alguns minutos para se familiarizar com a interface.

Caso ainda não tenha feito isso, siga [este link](../../envsetup/) antes de prosseguir.

Expand Down
34 changes: 17 additions & 17 deletions docs/hello_nextflow/03_hello_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,13 @@ If that worked for you, you're ready to learn how to assemble a multi-step workf
We're going to add a step to convert the greeting to uppercase.
To that end, we need to do three things:

- Define the command we'lre going to use to do the uppercase conversion.
- Define the command we're going to use to do the uppercase conversion.
- Write a new process that wraps the uppercasing command.
- Add the new process to the workflow and set it up to take the output of the `sayHello()` process as input.
- Call the new process in the workflow block and set it up to take the output of the `sayHello()` process as input.

### 1.1. Define the uppercasing command and test it in the terminal

To do the conversion of the greetings to uppercase, we're going to a classic UNIX tool called `tr` for 'text replacement', with the following syntax:
To do the conversion of the greetings to uppercase, we're going to use a classic UNIX tool called `tr` for 'text replacement', with the following syntax:

```bash title="Syntax"
tr '[a-z]' '[A-Z]'
Expand Down Expand Up @@ -338,7 +338,7 @@ Nextflow doesn't mind, so it doesn't matter.
This is where things could get a little tricky, because we need to be able to handle an arbitrary number of input files.
Specifically, we can't write the command up front, so we need to tell Nextflow how to compose it at runtime based on what inputs flow into the process.

In other words, if we have an input channel containing the item `[file1.txt, file2.txt, file3.txt]`, we need Nextflow to turn that into `cat file1.txt file2.txt file3.txt`.
In other words, if we have an input channel containing the element `[file1.txt, file2.txt, file3.txt]`, we need Nextflow to turn that into `cat file1.txt file2.txt file3.txt`.

Fortunately, Nextflow is quite happy to do that for us if we simply write `cat ${input_files}` in the script command.

Expand Down Expand Up @@ -368,7 +368,7 @@ In theory this should handle any arbitrary number of input files.

Some command-line tools require providing an argument (like `-input`) for each input file.
In that case, we would have to do a little bit of extra work to compose the command.
You can see an example of this in the 'Nextflow for Genomics' training course.
You can see an example of this in the [Nextflow for Genomics](https://training.nextflow.io/latest/nf4_science/genomics/) training course.

<!--[ADD LINK to note above] -->

Expand Down Expand Up @@ -427,13 +427,13 @@ We were only expecting one, but there are three.

And have a look at the contents of the final output file too:

```console title="COLLECTED-output.txt"
```console title="results/COLLECTED-output.txt"
Holà
```

Oh no. The collection step was run individually on each greeting, which is NOT what we wanted.

We need to do something to tell Nextflow explicitly that we want that third step to run on all the items in the channel output by `convertToUpper()`.
We need to do something to tell Nextflow explicitly that we want that third step to run on all the elements in the channel output by `convertToUpper()`.

### 2.3. Use an operator to collect the greetings into a single input

Expand Down Expand Up @@ -521,22 +521,22 @@ This time the third step was only called once!
Looking at the output of the `view()` statements, we see the following:

- Three `Before collect:` statements, one for each greeting: at that point the file paths are individual items in the channel.
- A single `After collect:` statement: the three file paths are now packaged into a single item.
- A single `After collect:` statement: the three file paths are now packaged into a single element.

Have a look at the contents of the final output file too:

```console title="COLLECTED-output.txt"
```console title="results/COLLECTED-output.txt"
BONJOUR
HELLO
HOLà
```

This time we have all three greetings in the final output file. Success!
This time we have all three greetings in the final output file. Success! Remove the optional view calls to make the next outputs less verbose.

!!! note

If you run this several times without `-resume`, you will see that the order of the greetings changes from one run to the next.
This shows you that the order in which items flow through the pipeline is not guaranteed to be consistent.
This shows you that the order in which elements flow through process calls is not guaranteed to be consistent, unless you make use of fair threading (check the [fair](https://www.nextflow.io/docs/latest/reference/process.html#fair) directive).

### Takeaway

Expand Down Expand Up @@ -742,14 +742,14 @@ Conveniently, Nextflow lets us add arbitrary code in the `script:` block of the

That means we can use the built-in `size()` function to get the number of files in the `input_files` array.

In the process block, make the following code change:
In the `collectGreetings` process block, make the following code change:

_Before:_

```groovy title="hello-workflow.nf" linenums="55"
script:
"""
cat ${input_files} > 'COLLECTED-${batch_id}-output.txt'
cat ${input_files} > 'COLLECTED-${batch_name}-output.txt'
"""
```

Expand All @@ -759,7 +759,7 @@ _After:_
script:
count_greetings = input_files.size()
"""
cat ${input_files} > 'COLLECTED-${batch_id}-output.txt'
cat ${input_files} > 'COLLECTED-${batch_name}-output.txt'
"""
```

Expand All @@ -777,14 +777,14 @@ _Before:_

```groovy title="hello-workflow.nf" linenums="52"
output:
path "COLLECTED-${batch_id}-output.txt"
path "COLLECTED-${batch_name}-output.txt"
```

_After:_

```groovy title="hello-workflow.nf" linenums="52"
output:
path "COLLECTED-${batch_id}-output.txt" , emit: outfile
path "COLLECTED-${batch_name}-output.txt" , emit: outfile
val count_greetings , emit: count
```

Expand All @@ -793,7 +793,7 @@ But as the saying goes, why not both?

### 4.2. Report the output at the end of the workflow

Now that we have two outputs coming out of the `collectGreetings` process, the `collectGreetings.out` output channel contains two 'tracks':
Now that we have two outputs coming out of the `collectGreetings` process, the `collectGreetings.out` output multi-channel contains two channels:

- `collectGreetings.out.outfile` contains the final output file
- `collectGreetings.out.count` contains the count of greetings
Expand Down
77 changes: 49 additions & 28 deletions docs/hello_nextflow/05_hello_containers.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ The `'<container>'` part is the URI address of the container image.
As an example, let's pull a container image that contains [cowpy](https://github.com/jeffbuttars/cowpy), a python implementation of a tool called `cowsay` that generates ASCII art to display arbitrary text inputs in a fun way.

There are various repositories where you can find published containers.
We used the [Seqera Containers](https://seqera.io/containers/) service to generate this Docker container from the `cowpy` Conda package: `'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'`.
We used the [Seqera Containers](https://seqera.io/containers/) service to generate this Docker container image from the `cowpy` Conda package: `'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'`.

Run the complete pull command:

Expand Down Expand Up @@ -163,7 +163,7 @@ You can also run a container interactively, which gives you a shell prompt insid

#### 1.3.1. Spin up the container

To run interactively, we just add `-it` to the `docker pull` command.
To run interactively, we just add `-it` to the `docker run` command.
Optionally, we can specify the shell we want to use inside the container by appending _e.g._ `/bin/bash` to the command.

```bash
Expand Down Expand Up @@ -200,7 +200,7 @@ For example, the tool documentation says we can change the character ('cowacter'
cowpy "Hello Containers" -c tux
```

Now the output shows the Linux penguin, Tux, instead of the default cow, because we specified `-c tux` parameter.
Now the output shows the Linux penguin, Tux, instead of the default cow, because we specified the `-c tux` parameter.

```console title="Output"
__________________
Expand Down Expand Up @@ -250,7 +250,7 @@ One way to do this is to **mount** a **volume** from the host system into the co
-v <outside_path>:<inside_path>
```

In our case `<outside_path>` will be the current working directory, so we can just use a dot (`.`), and `<outside_path>` is just a name we make up; let's call it `/data`.
In our case `<outside_path>` will be the current working directory, so we can just use a dot (`.`), and `<inside_path>` is just a name we make up; let's call it `/data`.

To mount a volume, we replace the paths and add the volume mounting argument to the docker run command as follows:

Expand All @@ -266,6 +266,8 @@ You can check that it works by listing the contents of `/data`:
ls /data
```

Depending on what part of this training you've done before, the output below my look slightly different.

```console title="Output"
demo-params.json hello-channels.nf hello-workflow.nf modules results
greetings.csv hello-modules.nf hello-world.nf nextflow.config work
Expand All @@ -291,9 +293,9 @@ This produces the desired ASCII art of a turkey rattling off our example greetin

```console title="Output"
_________
/ HOLà \
| HELLO |
\ BONJOUR /
/ Hello \
| Bonjour |
\ Holà /
---------
\ ,+*^^*+___+++_
\ ,*^^^^ )
Expand Down Expand Up @@ -502,6 +504,21 @@ ERROR ~ Error executing process > 'cowpy'

Caused by:
Process `cowpy` terminated with an error exit status (127)

Command executed:

cat COLLECTED-test-batch-output.txt | cowpy -c "turkey" > cowpy-COLLECTED-test-batch-output.txt

Command exit status:
127

Command output:
(empty)

Command error:
.command.sh: line 2: cowpy: command not found

(trimmed output)
```

This error code, `error exit status (127)` means the executable we asked for was not found.
Expand Down Expand Up @@ -530,7 +547,6 @@ _After:_
process cowpy {

publishDir 'containers/results', mode: 'copy'

container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
```

Expand Down Expand Up @@ -590,27 +606,32 @@ There were 3 greetings in this batch
You can find the cowpy'ed output in the `results` directory.

```console title="results/cowpy-COLLECTED-test-batch-output.txt"
_______
/ \
_________
/ HOLà \
| HELLO |
| HOLà |
| BONJOUR |
\ /
=======
\
\
\
\
,.
(_|,.
,' /, )_______ _
__j o``-' `.'-)'
(") \'
`-j |
`-._( /
|_\ |--^. /
/_]'|_| /_)_/
/_]' /_]'
\ BONJOUR /
---------
\ ,+*^^*+___+++_
\ ,*^^^^ )
\ _+* ^**+_
\ +^ _ _++*+_+++_, )
_+^^*+_ ( ,+*^ ^ \+_ )
{ ) ( ,( ,_+--+--, ^) ^\
{ (\@) } f ,( ,+-^ __*_*_ ^^\_ ^\ )
{:;-/ (_+*-+^^^^^+*+*<_ _++_)_ ) ) /
( / ( ( ,___ ^*+_+* ) < < \
U _/ ) *--< ) ^\-----++__) ) ) )
( ) _(^)^^)) ) )\^^^^^))^*+/ / /
( / (_))_^)) ) ) ))^^^^^))^^^)__/ +^^
( ,/ (^))^)) ) ) ))^^^^^^^))^^) _)
*+__+* (_))^) ) ) ))^^^^^^))^^^^^)____*^
\ \_)^)_)) ))^^^^^^^^^^))^^^^)
(_ ^\__^^^^^^^^^^^^))^^^^^^^)
^\___ ^\__^^^^^^))^^^^^^^^)\\
^^^^^\uuu/^^\uuu/^^^^\^\^\^\^\^\^\^\
___) >____) >___ ^\_\_\_\_\_\_\)
^^^//\\_^^//\\_^ ^(\_\_\_\)
^^^ ^^ ^^^ ^
```

You see that the character is saying all the greetings, just as it did when we ran the `cowpy` command on the `greetings.csv` file from inside the container.
Expand Down
4 changes: 2 additions & 2 deletions docs/nextflow_run/03_run_nf-core.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,9 +305,9 @@ Template features can be flexibly included or excluded at the time of creation,
5. On the Template features screen, turn **off**:

- `Use a GitHub repository`
- `Add Github CI tests`
- `Add GitHub CI tests`
- `Use reference genomes`
- `Add Github badges`
- `Add GitHub badges`
- `Include citations`
- `Include a gitpod environment`
- `Include GitHub Codespaces`
Expand Down