all.txt Infrastructure

⚠️ Disclaimer 🇩🇪: Dies ist ein read-only Repository / Mirror, das den Infrastructure-as-Code (IaC) und CI/CD-Setup enthält, der von &effect data solutions GmbH im Rahmen des Datenvorhabens zwischen all.txt und dem Civic Data Lab entwickelt wurde. Dieses Repository ist nicht dazu geeignet, direkt verwendet zu werden, da CI/CD-Code GitLab-spezifisch ist. Der Code kann dennoch für andere Projekte adaptiert werden und steht unter der MIT Lizenz (siehe LICENSE für mehr Informationen). Code und Dokumentation ist in Englisch. ⚠️

⚠️ Disclaimer 🇬🇧: This is a read-only repository / mirror of the repository containing the infrastructure-as-code (IaC) and CI/CD setup that was developed by &effect data solutions GmbH as part of the data project (Datenvorhaben) between all.txt and the Civic Data Lab. This repository cannot be used as is, as its CI/CD code is specific to GitLab where the original (private) repository resides. However, code can still be adapted for other projects and is under the MIT license (see LICENSE for details). ⚠️

Introduction

This repository contains the infrastructure code for the project. The infrastructure is composed of the following components: Provisioning and Configuration. The infrastructure is configured using Infrastructure-as-Code (IaC) principles.

The provisioning is done using Terraform. The configuration is done using Ansible. The infrastructure is deployed on Scaleway. The code is executed in a GitLab CI/CD pipeline.

Folder Structure and Functionality

Terraform (Provisioning)

variables.tf

The variables are defined in the variables.tf file. The variables are used to configure the Terraform code. The variables are used to configure the Scaleway provider and the database users. The variables are used in the main.tf file. Please configure all variables that are used in the variables.tf file and add them to the GitLab CI/CD pipeline variables.

main.tf

The provisioning is done using Terraform. We use the Scaleway provider to provision the infrastructure. The Terraform code is located in the main.tf file. We provision a network, a security group, a server, a floating IP, a Postgres server, and a Postgres database and users in this file.

• The server is used as Docker Swarm Leader. The server exposes all services running on the server using Traefik.
• The Postgres server is used as the centralized database for the services.
• The network is used to connect the server and the Postgres server.
• The security group is used to allow traffic to the server on port 22, 80, and 443. The Postgres server is only accessible from the server.
• The floating IP is used to access the server from the internet. If services are added to the server, you have to configure a DNS record to point to the floating IP.

cloud-init.yml

The cloud-init.yml file is used to configure the server. The file is used to configure the deploy user, the group and the SSH key. The file is used in the main.tf file.

inventory.tftpl

The inventory.tpl file is used to generate the Ansible inventory file. The file is used in the main.tf file. The file is used to generate the inventory file with the server IP addresses. All are appended to the inventory file.

Ansible (Configuration)

swarm.yml

The swarm.yml file is used to configure the server. The file consists of four tasks.

• The first tasks adds and installs the Docker prerequisites. The prerequisites are required to install and configure Docker.
• The second task adds and installs Docker. The task installs the latest version of Docker.
• The third task initializes the Docker Swarm cluster. The task initializes the server as a leader.
• The fourth task installs Traefik. The task installs Traefik as a reverse proxy. The task uses the traefik.compose.yml file to configure Traefik.

traefik.compose.yml

The traefik.compose.yml file is used to configure Traefik. The file is used in the swarm.yml file. The file configures Traefik to use the Docker backend and to use the Docker Swarm mode. Let's Encrypt is used to generate SSL certificates. The certificates are stored in the acme.json file. The file is mounted as a volume to the Traefik container.

Pipeline

.gitlab-ci.yml

The GitLab CI/CD pipeline is used to deploy the infrastructure. The pipeline is configured in the .gitlab-ci.yml file. Please see the next section for more information.

GitLab CI/CD Pipeline

The GitLab CI/CD pipeline is used to deploy the infrastructure. The pipeline is configured in the .gitlab-ci.yml file. The pipeline is composed of the following jobs: init, plan, apply, and configure. The first three jobs are used to provision the infrastructure. The last job is used to configure the infrastructure.

Configuration

The following variables need to be configured in the GitLab CI/CD pipeline. The variables can be configured in the GitLab console under Settings > CI/CD > Variables. Variables prefixed with TF_VAR_ are automatically used in the Terraform code.

• SSH_DEPLOY_KEY: The SSH private key to access the server.
• TF_VAR_scaleway_access_key: The Scaleway access key.
• TF_VAR_scaleway_secret_key: The Scaleway secret key.
• TF_VAR_rdb_user_***_password: A password for the Postgres user. Replace *** with the user name. Please configure a password for each user.

Job 1: Terraform Init and Validate

The first job is used to initialize Terraform and to validate the Terraform code. The job is used to check if the Terraform code is correct and all providers are available. The job is executed in the init stage.

Job 2: Terraform Plan

The second job is used to plan the Terraform code. The job is used to check if the Terraform code will create, update, or delete resources. The job is executed in the plan stage. After the job is executed, the plan is available in the GitLab console. You can evaluate the plan and decide if you want to apply the plan in the next job.

Job 3: Terraform Apply

The third job is used to apply the Terraform plan. The job is used to create, update, or delete the resources. The job is executed in the apply stage. The job is only executed if the plan is available from the previous job. The job is used to provision the actual infrastructure on Scaleway.

Job 4: Ansible Playbook

The fourth job is used to configure the infrastructure. The job is used to configure the server and to install the necessary software. The job is executed in the configure stage. The job is only executed if the infrastructure is provisioned successfully. The job relies on the inventory.tpl file to generate the Ansible inventory file. The job uses the swarm.yml file to configure the server.

FAQ

How to access the server?

Please use the floating IP to access the server. The floating IP is configured in the Scaleway console. The server is accessible on port 22. Use SSH to access the server. The deploy user has the necessary permissions to access the server and to manage Docker. The root user has all permissions.

# connect to server
ssh deploy@XX.XXX.XX.XXX -i ~/.ssh/private_key

How to update the server?

Please use the root user to update the server. Please use the following commands to update the server.

sudo apt-get update
sudo apt-get upgrade

If necessary reboot the server. The server will be available shortly after the reboot.

sudo reboot

How to access the Postgres server?

The Postgres server is only accessible from the server. Please use port forwarding to access the Postgres server. Please configure a user in the main.tf file to access the Postgres server.

# open tunnel to Postgres server
ssh -f deploy@51.158.117.130 -i ~/.ssh/private_key -L 5432:172.16.16.2:5432 -N