Skip to content

Enable local use via multi devcontainer #533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 25 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c86aa4f
Small fixes to Hello World (#528)
Desk3505 Feb 28, 2025
e6d3f13
Add NF4 RNAseq mini-course v0 (#526)
vdauwera Feb 28, 2025
523ba38
Oopsies
vdauwera Feb 28, 2025
5c94632
Improved clarity and consistency in 'get started' call to action
vdauwera Feb 28, 2025
761c257
Try for multi devcontainer setup
pinin4fjords Mar 3, 2025
6818b8f
Add other file
pinin4fjords Mar 3, 2025
9ab979d
rename
pinin4fjords Mar 3, 2025
d02b178
delete original
pinin4fjords Mar 3, 2025
73718ea
Reapply mount settings for local
pinin4fjords Mar 3, 2025
6c5e12d
Fix local devcontainers
pinin4fjords Mar 3, 2025
a3c7f34
Symlink codespaces as default
pinin4fjords Mar 3, 2025
198bd5d
Add complete workflow file before genomics modules part
pinin4fjords Mar 3, 2025
c493c62
Finish genomics modules component
pinin4fjords Mar 3, 2025
aa99365
prettier
pinin4fjords Mar 3, 2025
00fd5c3
update mkdocs config
pinin4fjords Mar 3, 2025
7c0f8d7
Fix minor issues
pinin4fjords Mar 3, 2025
4e74bcd
Leave workflow file solution in main dir as previous
pinin4fjords Mar 3, 2025
8b288be
Typos
ewels Mar 4, 2025
6276c6b
Don't use $it
ewels Mar 4, 2025
0351bfe
Fix admonition text inclusion
ewels Mar 5, 2025
53af0d9
Kill the squiggles
ewels Mar 5, 2025
6e53d14
Merge pull request #538 from ewels/training-small-fixes
ewels Mar 5, 2025
011340e
Merge pull request #531 from nextflow-io/gvda-patch_rnaseq
ewels Mar 5, 2025
99df8a2
Merge pull request #534 from nextflow-io/nf_test_genomics
pinin4fjords Mar 5, 2025
a5077b5
Merge branch 'master' into multi_devcontainer
pinin4fjords Mar 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .devcontainer/codespaces/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
{
"name": "codespaces",
"build": {
"dockerfile": "Dockerfile",
"context": ".."
},
"remoteUser": "gitpod",
"features": {
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}
},
"workspaceFolder": "/workspaces/training",
"workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/training,type=bind", // Used for local devcontainers, ignored in Codespaces.
"remoteEnv": {
"NXF_HOME": "/workspaces/.nextflow",
"HOST_PROJECT_PATH": "/workspaces/training" //Codespaces and local devcontainers will both work with this.
},
"postCreateCommand": "if [ -z \"$CODESPACES\" ]; then echo \"Local Development\"; else echo \"Codespaces Development\"; fi",
// Configure tool-specific properties.
"customizations": {
// Configure properties specific to VS Code.
"vscode": {
// Set *default* container specific settings.json values on container create.
"settings": {
"python.defaultInterpreterPath": "/opt/conda/bin/python"
},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"nf-core.nf-core-extensionpack",
"nextflow.nextflow",
"codezombiech.gitignore"
]
}
},
"portsAttributes": {
"3000": {
"label": "Application",
"onAutoForward": "openPreview"
}
}
}
43 changes: 0 additions & 43 deletions .devcontainer/devcontainer.json

This file was deleted.

1 change: 1 addition & 0 deletions .devcontainer/devcontainer.json
45 changes: 45 additions & 0 deletions .devcontainer/local/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
{
"name": "local",
"build": {
"dockerfile": "Dockerfile",
"context": ".."
},
"remoteUser": "gitpod",
"features": {
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}
},
"workspaceFolder": "${localWorkspaceFolder}",
"workspaceMount": "source=${localWorkspaceFolder},target=${localWorkspaceFolder},type=bind",
"remoteEnv": {
"NXF_HOME": "/workspaces/training/.nextflow",
"HOST_PROJECT_PATH": "/workspaces/training"
},
"postCreateCommand": "sudo mkdir -p /workspaces && sudo ln -s ${localWorkspaceFolder} /workspaces/training && if [ -z \"$CODESPACES\" ]; then echo \"Local Development\"; else echo \"Codespaces Development\"; fi",
"postAttachCommand": "cd /workspaces/training",
// Configure tool-specific properties.
"customizations": {
// Configure properties specific to VS Code.
"vscode": {
// Set *default* container specific settings.json values on container create.
"settings": {
"python.defaultInterpreterPath": "/opt/conda/bin/python",
"terminal.integrated.cwd": "/workspaces/training",
"terminal.integrated.defaultLocation": "workspace"
},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"nf-core.nf-core-extensionpack",
"nextflow.nextflow",
"codezombiech.gitignore"
]
}
},
"portsAttributes": {
"3000": {
"label": "Application",
"onAutoForward": "openPreview"
}
}
}
8 changes: 4 additions & 4 deletions docs/hello_nextflow/01_hello_world.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ process sayHello {
}
```

This a very minimal process definition that just contains an `output` definition and the `script` to execute.
This is a very minimal process definition that just contains an `output` definition and the `script` to execute.

The `output` definition includes the `path` qualifier, which tells Nextflow this should be handled as a path (includes both directory paths and files).
Another common qualifier is `val`.
Expand Down Expand Up @@ -172,7 +172,7 @@ workflow {
}
```

This a very minimal **workflow** definition.
This is a very minimal **workflow** definition.
In a real-world pipeline, the workflow typically contains multiple calls to **processes** connected by **channels**, and the processes expect one or more variable **input(s)**.

You'll learn how to add variable inputs later in this training module; and you'll learn how to add more processes and connect them by channels in Part 3 of this course.
Expand Down Expand Up @@ -479,7 +479,7 @@ In the process block, make the following code change:

_Before:_

```groovy title="hello-channels.nf" linenums="6"
```groovy title="hello-world.nf" linenums="6"
process sayHello {

publishDir 'results', mode: 'copy'
Expand All @@ -490,7 +490,7 @@ process sayHello {

_After:_

```groovy title="hello-channels.nf" linenums="6"
```groovy title="hello-world.nf" linenums="6"
process sayHello {

publishDir 'results', mode: 'copy'
Expand Down
37 changes: 25 additions & 12 deletions docs/hello_nextflow/02_hello_channels.md
Original file line number Diff line number Diff line change
Expand Up @@ -522,7 +522,7 @@ Here we added the operator on the next line for readability, but you can add ope

#### 3.2.2. Add `view()` to inspect channel contents

We could run this right away to test if it works, but while we're at it, we're also going to add a couple of [`view()`](https://www.nextflow.io/docs/latest/reference/operator.html#view) directives, which allow us to inspect the contents of a channel.
We could run this right away to test if it works, but while we're at it, we're also going to add a couple of [`view()`](https://www.nextflow.io/docs/latest/reference/operator.html#view) operators, which allow us to inspect the contents of a channel.
You can think of `view()` as a debugging tool, like a `print()` statement in Python, or its equivalent in other languages.

In the workflow block, make the following code change:
Expand All @@ -540,12 +540,25 @@ _After:_
```groovy title="hello-channels.nf" linenums="31"
// create a channel for inputs
greeting_ch = Channel.of(greetings_array)
.view { "Before flatten: $it" }
.view { greeting -> "Before flatten: $greeting" }
.flatten()
.view { "After flatten: $it" }
.view { greeting -> "After flatten: $greeting" }
```

Here `$it` is an implicit variable that represents each individual item loaded in a channel.
We are using an operator _closure_ here - the curly brackets.
This code executes for each item in the channel.
We define a temporary variable for the inner value, here called `greeting` (it could be anything).
This variable is only used within the scope of that closure.

In this example, `$greeting` represents each individual item loaded in a channel.

!!! note "Note on `$it`"

In some pipelines you may see a special variable called `$it` used inside operator closures.
This is an _implicit_ variable that allows a short-hand access to the inner variable,
without needing to define it with a `->`.

We prefer to be explicit to aid code clarity, as such the `$it` syntax is discouraged and will slowly be phased out of the Nextflow language.

#### 3.2.3. Run the workflow

Expand Down Expand Up @@ -723,9 +736,9 @@ _After:_
```groovy title="hello-channels.nf" linenums="31"
// create a channel for inputs from a CSV file
greeting_ch = Channel.fromPath(params.greeting)
.view { "Before splitCsv: $it" }
.view { csv -> "Before splitCsv: $csv" }
.splitCsv()
.view { "After splitCsv: $it" }
.view { csv -> "After splitCsv: $csv" }
```

As you can see, we also include before/after view statements while we're at it.
Expand Down Expand Up @@ -787,7 +800,7 @@ This is what the syntax looks like:

This means 'for each element in the channel, take the first of any items it contains'.

So let's apply that to our CVS parsing.
So let's apply that to our CSV parsing.

#### 4.3.1. Apply `map()` to the channel

Expand All @@ -798,21 +811,21 @@ _Before:_
```groovy title="hello-channels.nf" linenums="31"
// create a channel for inputs from a CSV file
greeting_ch = Channel.fromPath(params.greeting)
.view { "Before splitCsv: $it" }
.view { csv -> "Before splitCsv: $csv" }
.splitCsv()
.view { "After splitCsv: $it" }
.view { csv -> "After splitCsv: $csv" }
```

_After:_

```groovy title="hello-channels.nf" linenums="31"
// create a channel for inputs from a CSV file
greeting_ch = Channel.fromPath(params.greeting)
.view { "Before splitCsv: $it" }
.view { csv -> "Before splitCsv: $csv" }
.splitCsv()
.view { "After splitCsv: $it" }
.view { csv -> "After splitCsv: $csv" }
.map { item -> item[0] }
.view { "After map: $it" }
.view { csv -> "After map: $csv" }
```

Once again we include another `view()` call to confirm that the operator does what we expect.
Expand Down
8 changes: 3 additions & 5 deletions docs/hello_nextflow/03_hello_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -485,8 +485,8 @@ _After:_
collectGreetings(convertToUpper.out.collect())

// optional view statements
convertToUpper.out.view { "Before collect: $it" }
convertToUpper.out.collect().view { "After collect: $it" }
convertToUpper.out.view { greeting -> "Before collect: $greeting" }
convertToUpper.out.collect().view { greeting -> "After collect: $greeting" }
}
```

Expand Down Expand Up @@ -816,11 +816,9 @@ _After:_
collectGreetings(convertToUpper.out.collect(), params.batch)

// emit a message about the size of the batch
collectGreetings.out.count.view { "There were $it greetings in this batch" }
collectGreetings.out.count.view { num_greetings -> "There were $num_greetings greetings in this batch" }
```

Here we are using `$it` in the same way we did earlier, as an implicit variable to access the contents of the channel.

!!! note

There are a few other ways we could achieve a similar result, including some more elegant ones like the `count()` operator, but this allows us to show how to handle multiple outputs, which is what we care about.
Expand Down
6 changes: 3 additions & 3 deletions docs/hello_nextflow/05_hello_containers.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ You can see that the filesystem inside the container is different from the files
When you run a container, it is isolated from the host system by default.
This means that the container can't access any files on the host system unless you explicitly allow it to do so.

You will learn how to do that in a minute.
You will learn how to do that in a minute.

#### 1.3.2. Run the desired tool command(s)

Expand Down Expand Up @@ -434,7 +434,7 @@ _Before:_
collectGreetings(convertToUpper.out.collect(), params.batch)

// emit a message about the size of the batch
collectGreetings.out.count.view{ "There were $it greetings in this batch" }
collectGreetings.out.count.view{ num_greetings -> "There were $num_greetings greetings in this batch" }
```

_After:_
Expand All @@ -444,7 +444,7 @@ _After:_
collectGreetings(convertToUpper.out.collect(), params.batch)

// emit a message about the size of the batch
collectGreetings.out.count.view{ "There were $it greetings in this batch" }
collectGreetings.out.count.view{ num_greetings -> "There were $num_greetings greetings in this batch" }

// generate ASCII art of the greetings with cowpy
cowpy(collectGreetings.out.outfile, params.character)
Expand Down
2 changes: 1 addition & 1 deletion docs/hello_nextflow/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The rise of big data has made it increasingly necessary to be able to analyze an

During this training, you will be introduced to Nextflow in a series of complementary hands-on workshops.

Let's get started! Click on the "Open in GitHub Codespaces" button below.
Let's get started! Click on the "Open in GitHub Codespaces" button below to launch the training environment (preferably in a separate tab), then read on while it loads.

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/nextflow-io/training?quickstart=1&ref=master)

Expand Down
Loading