From 734d6ab177333101194c34cbb9b749ef6dd22ce5 Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Mon, 14 Apr 2025 10:04:19 +0200 Subject: [PATCH 1/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index c63c27e7..6557ba70 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -14,6 +14,7 @@ Test code in [automation lecture repository](https://gitlab-sim.informatik.uni-s - Edit in pipeline editor -> Visualize - Settings -> CI/CD -> Runners -> Specific runners - URL and Token; we will need this in a minute + - Select `Run untagged jobs` (optional, but can make things simpler for now) ## Inspect bwCloud @@ -22,24 +23,28 @@ Test code in [automation lecture repository](https://gitlab-sim.informatik.uni-s - I have already set up a VM. What I did: - Add public SSH key - Instances -> Launch instance - - Ubuntu 22.04 + - Ubuntu 24.04 - Flavor: m1.small - `sudo apt update && sudo apt -y upgrade` - `sudo apt install -y docker.io` - VM is up and running, connect to it: `ssh ubuntu@` +You can get the IP from the [Instances view](https://portal.bw-cloud.org/project/instances/). Note that there are two addresses here: an IPv4 (decimal) and an IPv6 (hexadecimal) address. New VMs on bwCloud only support IPv6 networks, so you need the second address. + ## Setup GitLab Runner - Install GitLab Runner via Docker: ```bash sudo docker run -d --name gitlab-runner --restart always \ + --network host \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest ``` - `docker run -d --name gitlab-runner --restart always` runs the container in the background (`-d` means detached) names it `gitlab-runner` and makes sure that it always runs. The container is automatically restarted once it stops/crashes. If you want to stop the container, you have to stop it manually (`docker container stop`). + - `--network host` tells Docker to use the host network stack and is needed on bwCloud VMs, which only support IPv6. - `-v /srv/gitlab-runner/config:/etc/gitlab-runner` mounts the directory `/srv/gitlab-runner/config` into the container. - `-v /var/run/docker.sock:/var/run/docker.sock` mounts important Docker files into the container such that the container can start other containers (for pipelines). - `gitlab/gitlab-runner:latest` is the GitLab Runner image used from Docker Hub. @@ -51,8 +56,18 @@ Test code in [automation lecture repository](https://gitlab-sim.informatik.uni-s ## Register Runner -- `sudo docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register` - - URL: `https://gitlab-sim.informatik.uni-stuttgart.de/` +You can register a runner using the following command. Notice again the `--network host` option, which tells Docker to use the host network stack. This is an important workaround for the fact that bwCloud only supports IPv6 networks: + +```bash +sudo docker run --rm -it \ + --network host \ + -v /srv/gitlab-runner/config:/etc/gitlab-runner \ + gitlab/gitlab-runner register \ + --url https://gitlab-sim.informatik.uni-stuttgart.de/ +``` + +- Enter the following details: + - URL: (press Enter to confirm) - Token: see above - Description: `SSE Automation Demo Runner` - No tags, no maintenance note From e2f4662617d10bfc0aa256885f760b7753277cc0 Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Tue, 15 Apr 2025 14:25:53 +0200 Subject: [PATCH 2/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index 6557ba70..e85819c4 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -66,7 +66,7 @@ sudo docker run --rm -it \ --url https://gitlab-sim.informatik.uni-stuttgart.de/ ``` -- Enter the following details: +- Enter the following details: - URL: (press Enter to confirm) - Token: see above - Description: `SSE Automation Demo Runner` From a555f4d4ceacd89d43bd70266ac3260e164847c6 Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Wed, 16 Apr 2025 17:07:30 +0200 Subject: [PATCH 3/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index e85819c4..69f904c1 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -31,20 +31,44 @@ Test code in [automation lecture repository](https://gitlab-sim.informatik.uni-s You can get the IP from the [Instances view](https://portal.bw-cloud.org/project/instances/). Note that there are two addresses here: an IPv4 (decimal) and an IPv6 (hexadecimal) address. New VMs on bwCloud only support IPv6 networks, so you need the second address. +## Workarounds for IPv6 + +New bwCloud VMs only support IPv6 by default. Asking for IPv4 for a specific VM might be possible via the [helpdesk](https://bw-cloud.org/q/t). + +A few workarounds are needed to make the GitLab runner work in this IPv6-only environment, especially since [`registry.gitlab.com` does not support IPv6](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/18058) and the Docker support for IPv6 needs to be expliticly configured. + +First, we need to tell Docker to use the host network and DNS. We also need to replace the helper image with [the one from Docker Hub](https://hub.docker.com/r/gitlab/gitlab-runner-helper/tags?name=x86_64-v17.10.1). Edit the `[runners.docker]` section in `/srv/gitlab-runner/config/config.toml`: + +```toml + volumes = ["/etc/resolv.conf:/etc/resolv.conf:ro", "/srv/gitlab-runner/custom-hosts:/etc/hosts:ro", "/cache"] + helper_image = "gitlab/gitlab-runner-helper:x86_64-v17.10.1" + network_mode = "host" +``` + +While you could pass `--network host` to `docker run`, setting this system-wide makes it easier to also start the job containers with the same settings. + +Whenever changing this configuration file, we need to restart the runner (e.g., by restarting the respective container). + +We also need to hard-code the IPv6 address that corresponds to the GitLab instance domain. Add the following line in the `/srv/gitlab-runner/custom-hosts`: + +``` +2001:7c0:2015:216::19 gitlab-sim.informatik.uni-stuttgart.de +``` + +You can get this address by running `dig aaaa gitlab-sim.informatik.uni-stuttgart.de`. + ## Setup GitLab Runner - Install GitLab Runner via Docker: ```bash sudo docker run -d --name gitlab-runner --restart always \ - --network host \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest ``` - `docker run -d --name gitlab-runner --restart always` runs the container in the background (`-d` means detached) names it `gitlab-runner` and makes sure that it always runs. The container is automatically restarted once it stops/crashes. If you want to stop the container, you have to stop it manually (`docker container stop`). - - `--network host` tells Docker to use the host network stack and is needed on bwCloud VMs, which only support IPv6. - `-v /srv/gitlab-runner/config:/etc/gitlab-runner` mounts the directory `/srv/gitlab-runner/config` into the container. - `-v /var/run/docker.sock:/var/run/docker.sock` mounts important Docker files into the container such that the container can start other containers (for pipelines). - `gitlab/gitlab-runner:latest` is the GitLab Runner image used from Docker Hub. @@ -60,7 +84,6 @@ You can register a runner using the following command. Notice again the `--netwo ```bash sudo docker run --rm -it \ - --network host \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ gitlab/gitlab-runner register \ --url https://gitlab-sim.informatik.uni-stuttgart.de/ From f4e4336559aa09455be278cf726b9860d3ec9615 Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Wed, 16 Apr 2025 17:32:09 +0200 Subject: [PATCH 4/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 61 +++++++++++++++-------------- 1 file changed, 32 insertions(+), 29 deletions(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index 69f904c1..2567422d 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -25,38 +25,12 @@ Test code in [automation lecture repository](https://gitlab-sim.informatik.uni-s - Instances -> Launch instance - Ubuntu 24.04 - Flavor: m1.small - - `sudo apt update && sudo apt -y upgrade` - - `sudo apt install -y docker.io` - VM is up and running, connect to it: `ssh ubuntu@` +- Apply updates: `sudo apt update && sudo apt -y upgrade` +- Install Docker: `sudo apt install -y docker.io` You can get the IP from the [Instances view](https://portal.bw-cloud.org/project/instances/). Note that there are two addresses here: an IPv4 (decimal) and an IPv6 (hexadecimal) address. New VMs on bwCloud only support IPv6 networks, so you need the second address. -## Workarounds for IPv6 - -New bwCloud VMs only support IPv6 by default. Asking for IPv4 for a specific VM might be possible via the [helpdesk](https://bw-cloud.org/q/t). - -A few workarounds are needed to make the GitLab runner work in this IPv6-only environment, especially since [`registry.gitlab.com` does not support IPv6](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/18058) and the Docker support for IPv6 needs to be expliticly configured. - -First, we need to tell Docker to use the host network and DNS. We also need to replace the helper image with [the one from Docker Hub](https://hub.docker.com/r/gitlab/gitlab-runner-helper/tags?name=x86_64-v17.10.1). Edit the `[runners.docker]` section in `/srv/gitlab-runner/config/config.toml`: - -```toml - volumes = ["/etc/resolv.conf:/etc/resolv.conf:ro", "/srv/gitlab-runner/custom-hosts:/etc/hosts:ro", "/cache"] - helper_image = "gitlab/gitlab-runner-helper:x86_64-v17.10.1" - network_mode = "host" -``` - -While you could pass `--network host` to `docker run`, setting this system-wide makes it easier to also start the job containers with the same settings. - -Whenever changing this configuration file, we need to restart the runner (e.g., by restarting the respective container). - -We also need to hard-code the IPv6 address that corresponds to the GitLab instance domain. Add the following line in the `/srv/gitlab-runner/custom-hosts`: - -``` -2001:7c0:2015:216::19 gitlab-sim.informatik.uni-stuttgart.de -``` - -You can get this address by running `dig aaaa gitlab-sim.informatik.uni-stuttgart.de`. - ## Setup GitLab Runner - Install GitLab Runner via Docker: @@ -84,6 +58,7 @@ You can register a runner using the following command. Notice again the `--netwo ```bash sudo docker run --rm -it \ + --network host \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ gitlab/gitlab-runner register \ --url https://gitlab-sim.informatik.uni-stuttgart.de/ @@ -92,7 +67,7 @@ sudo docker run --rm -it \ - Enter the following details: - URL: (press Enter to confirm) - Token: see above - - Description: `SSE Automation Demo Runner` + - Name/Description: `SSE Automation Demo Runner` - No tags, no maintenance note - Executor: `docker` - Default Docker image: `alpine:latest` (used for pipelines that do not specify any Docker image themselves, can be overwritten in configuration of pipeline) @@ -100,6 +75,34 @@ sudo docker run --rm -it \ - Verify that there is now a runner in the repo settings - Verify that pipeline now ran +The `--network host` is needed in this IPv6-only VM (see section below). + - More information: - [Executors and their abilities](https://docs.gitlab.com/runner/executors/) - [Registering runners](https://docs.gitlab.com/runner/register/index.html#docker). + +## Workarounds for IPv6 + +New bwCloud VMs only support IPv6 by default. Asking for IPv4 for a specific VM might be possible via the [helpdesk](https://bw-cloud.org/q/t). + +A few workarounds are needed to make the GitLab runner work in this IPv6-only environment, especially since [`registry.gitlab.com` does not support IPv6](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/18058) and the Docker support for IPv6 needs to be expliticly configured. + +First, we need to tell Docker to use the host network and DNS. We also need to replace the helper image with [the one from Docker Hub](https://hub.docker.com/r/gitlab/gitlab-runner-helper/tags?name=x86_64-v17.10.1) (depends on the GitLab version. You can start without this setting, and see which image GitLab is trying to pull). Edit the `[runners.docker]` section in `/srv/gitlab-runner/config/config.toml`: + +```toml + volumes = ["/etc/resolv.conf:/etc/resolv.conf:ro", "/srv/gitlab-runner/custom-hosts:/etc/hosts:ro", "/cache"] + helper_image = "gitlab/gitlab-runner-helper:x86_64-v17.10.1" + network_mode = "host" +``` + +While we already pass `--network host` to `docker run`, setting this system-wide makes it easier to also start the job containers with the same settings. + +Whenever changing this configuration file, we need to restart the runner (e.g., by restarting the respective container). + +We also need to hard-code the IPv6 address that corresponds to the GitLab instance domain. Add the following line in the `/srv/gitlab-runner/custom-hosts` (create the file): + +``` +2001:7c0:2015:216::19 gitlab-sim.informatik.uni-stuttgart.de +``` + +You can get this address by running `dig aaaa gitlab-sim.informatik.uni-stuttgart.de`. From 593dfcf9b18c7f08e5725649d680eb56e4232c5c Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Wed, 16 Apr 2025 18:49:17 +0200 Subject: [PATCH 5/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index 2567422d..ffe7ae87 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -59,6 +59,7 @@ You can register a runner using the following command. Notice again the `--netwo ```bash sudo docker run --rm -it \ --network host \ + -e GODEBUG="netdns=go+ipv6" \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ gitlab/gitlab-runner register \ --url https://gitlab-sim.informatik.uni-stuttgart.de/ @@ -75,7 +76,7 @@ sudo docker run --rm -it \ - Verify that there is now a runner in the repo settings - Verify that pipeline now ran -The `--network host` is needed in this IPv6-only VM (see section below). +The `--network host` and `-e GODEBUG="netdns=go+ipv6"` are needed in this IPv6-only VM (see section below). - More information: - [Executors and their abilities](https://docs.gitlab.com/runner/executors/) From 9f72004957ad1b8492198e4b9f9ba98368fd95ba Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Wed, 16 Apr 2025 18:56:06 +0200 Subject: [PATCH 6/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index ffe7ae87..f55f775a 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -91,19 +91,9 @@ A few workarounds are needed to make the GitLab runner work in this IPv6-only en First, we need to tell Docker to use the host network and DNS. We also need to replace the helper image with [the one from Docker Hub](https://hub.docker.com/r/gitlab/gitlab-runner-helper/tags?name=x86_64-v17.10.1) (depends on the GitLab version. You can start without this setting, and see which image GitLab is trying to pull). Edit the `[runners.docker]` section in `/srv/gitlab-runner/config/config.toml`: ```toml - volumes = ["/etc/resolv.conf:/etc/resolv.conf:ro", "/srv/gitlab-runner/custom-hosts:/etc/hosts:ro", "/cache"] helper_image = "gitlab/gitlab-runner-helper:x86_64-v17.10.1" network_mode = "host" ``` While we already pass `--network host` to `docker run`, setting this system-wide makes it easier to also start the job containers with the same settings. -Whenever changing this configuration file, we need to restart the runner (e.g., by restarting the respective container). - -We also need to hard-code the IPv6 address that corresponds to the GitLab instance domain. Add the following line in the `/srv/gitlab-runner/custom-hosts` (create the file): - -``` -2001:7c0:2015:216::19 gitlab-sim.informatik.uni-stuttgart.de -``` - -You can get this address by running `dig aaaa gitlab-sim.informatik.uni-stuttgart.de`. From 9e4eeee610ab9136ec316922ba2c6ad3d83dcd15 Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Wed, 16 Apr 2025 18:56:28 +0200 Subject: [PATCH 7/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index f55f775a..c9b550dc 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -14,7 +14,7 @@ Test code in [automation lecture repository](https://gitlab-sim.informatik.uni-s - Edit in pipeline editor -> Visualize - Settings -> CI/CD -> Runners -> Specific runners - URL and Token; we will need this in a minute - - Select `Run untagged jobs` (optional, but can make things simpler for now) + - Select `Run untagged jobs` ## Inspect bwCloud From f6cb2b53ec65e4ee22565a6bc3300f9d5ab8f33a Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Wed, 16 Apr 2025 18:59:45 +0200 Subject: [PATCH 8/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index c9b550dc..7242ec0e 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -84,11 +84,11 @@ The `--network host` and `-e GODEBUG="netdns=go+ipv6"` are needed in this IPv6-o ## Workarounds for IPv6 -New bwCloud VMs only support IPv6 by default. Asking for IPv4 for a specific VM might be possible via the [helpdesk](https://bw-cloud.org/q/t). +New bwCloud VMs only support IPv6 by default. Asking for IPv4 for a specific VM might be possible via the [helpdesk](https://bw-cloud.org/q/t). A few workarounds are needed to make the GitLab runner work in this IPv6-only environment. -A few workarounds are needed to make the GitLab runner work in this IPv6-only environment, especially since [`registry.gitlab.com` does not support IPv6](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/18058) and the Docker support for IPv6 needs to be expliticly configured. +First, we need to start Docker with `-e GODEBUG="netdns=go+ipv6"`. This is related to Go prioritizing IPv4 connections. -First, we need to tell Docker to use the host network and DNS. We also need to replace the helper image with [the one from Docker Hub](https://hub.docker.com/r/gitlab/gitlab-runner-helper/tags?name=x86_64-v17.10.1) (depends on the GitLab version. You can start without this setting, and see which image GitLab is trying to pull). Edit the `[runners.docker]` section in `/srv/gitlab-runner/config/config.toml`: +We also need to tell Docker to use the host network stack. We also need to replace the helper image with [the one from Docker Hub](https://hub.docker.com/r/gitlab/gitlab-runner-helper/tags?name=x86_64-v17.10.1) (depends on the GitLab version), since [`registry.gitlab.com` does not support IPv6](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/18058). You can start without this setting, and see which image GitLab is trying to pull). Edit the `[runners.docker]` section in `/srv/gitlab-runner/config/config.toml`: ```toml helper_image = "gitlab/gitlab-runner-helper:x86_64-v17.10.1" From d18db5ab7e103a6b311b1977492f9339423b680b Mon Sep 17 00:00:00 2001 From: Gerasimos Chourdakis Date: Wed, 16 Apr 2025 19:18:13 +0200 Subject: [PATCH 9/9] Update gitlab_ci_demo.md --- 05_testing_and_ci/gitlab_ci_demo.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/05_testing_and_ci/gitlab_ci_demo.md b/05_testing_and_ci/gitlab_ci_demo.md index 7242ec0e..cbdb3405 100644 --- a/05_testing_and_ci/gitlab_ci_demo.md +++ b/05_testing_and_ci/gitlab_ci_demo.md @@ -37,12 +37,15 @@ You can get the IP from the [Instances view](https://portal.bw-cloud.org/project ```bash sudo docker run -d --name gitlab-runner --restart always \ + --network host \ + -e GODEBUG="netdns=go+ipv6" \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:latest ``` - `docker run -d --name gitlab-runner --restart always` runs the container in the background (`-d` means detached) names it `gitlab-runner` and makes sure that it always runs. The container is automatically restarted once it stops/crashes. If you want to stop the container, you have to stop it manually (`docker container stop`). + - The `--network host` and `-e GODEBUG="netdns=go+ipv6"` are needed in this IPv6-only VM (see section below). - `-v /srv/gitlab-runner/config:/etc/gitlab-runner` mounts the directory `/srv/gitlab-runner/config` into the container. - `-v /var/run/docker.sock:/var/run/docker.sock` mounts important Docker files into the container such that the container can start other containers (for pipelines). - `gitlab/gitlab-runner:latest` is the GitLab Runner image used from Docker Hub. @@ -76,8 +79,6 @@ sudo docker run --rm -it \ - Verify that there is now a runner in the repo settings - Verify that pipeline now ran -The `--network host` and `-e GODEBUG="netdns=go+ipv6"` are needed in this IPv6-only VM (see section below). - - More information: - [Executors and their abilities](https://docs.gitlab.com/runner/executors/) - [Registering runners](https://docs.gitlab.com/runner/register/index.html#docker).