Startup Infra for Small Self-hosted Projects

This repository provides Ansible playbooks to set up a minimal infrastructure for a simple self-hosted application. Ideal for small hobby projects. I made this repository a place to Learn about DevOps and Cloud Infrastructure. You have all the tutorial you need to get started.

Features:

• docker-swarm
• Caddy
• Portainer

Development Environment Setup

This project provides a convenient dispatcher command, ibt, to help you set up your local development environment and manage project tools. Additionally, a Gitpod configuration is available for a cloud-based, ready-to-use environment.

Unified Tools Bash script: ibt

The ibt command (Infra Bootstrap Tools) is a shell function that provides a unified interface to the main project scripts with subcommands and auto-completion support.

Subcommands:

• ibt setup [tool ...] — Install required tools and dependencies (see below)
• ibt stacks [args ...] — Manage and run infrastructure stacks
• ibt tools [args ...] — Use Docker-based aliases for Ansible, AWS CLI, etc.

Auto-completion:

Tab-completion is available for subcommands and for the setup tool list (e.g., ibt setup [TAB]).

To enable ibt in your shell:

source ./bin/bash/ibt.sh

You can add this line to your ~/.bashrc or ~/.bash_profile for persistence.

Example usage:

# Install pre-commit and Ansible
ibt setup pre-commit ansible

# List available stacks
ibt stacks list

# Use Docker-based Ansible
ibt tools dasb --version

Gitpod

Alternatively, you can use Gitpod to get a pre-configured development environment in your browser. Click the button below to get started:

🚀 Quick Start

Got your prerequisites and secrets configured?

Deploy your entire infrastructure with a single command:

make up

This command will:

• Provision infrastructure on DigitalOcean using Terraform.
• Install Docker on all hosts.
• Initialize a Docker Swarm cluster.
• Deploy Caddy and Portainer applications.

To tear down the infrastructure:

make down

For a detailed step-by-step guide, including prerequisites and secret management options, please refer to our Full Getting Started Guide.

To understand the underlying Ansible concepts, check out Understanding Ansible Concepts.

Articles and Tutorials

The articles/tutorials are divided into sections.

• Introduction -> What are the tools to manage infrastructure? Perfect to learn the basis.
• How-tos -> Good takeaway from this project - Answers many questions you could encounter in the future
• Deepening Understanding -> Learn more about each application used in this setup (Portainer, Graphana, Caddy, etc.)
• Local Experimentation & Advanced Topics: Deeper dives into specific setups and components.

I used DigitalOcean for experiments because it is cheap, but any cloud provider should work, as we are mainly playing with Virtual Machines. You can even get $200 DigitalOcean free credit when starting

• Don't forget to delete everything after a tutorial if you don't want to add unnecessary costs

Tools Introduction

• WIP: 📚 1: What is Terraform and why you might need it.
• 📚 2: What is Terraform Cloud and why you might need it.
• WIP: 📚 3: What is Github Action and why you might need it.
• WIP: 📚 4: What is Ansible and why you might need it.
• WIP: 📚 5: What is Ansible AWX and why you might need it.

Learn the Tools

• 🌍 How to configure GitHub Environments with Terraform?
• 🏭 How to provision VM on Digital Ocean with Terraform?
• 🔏 How to create SSH keys with Terraform?
• 🗺️ How to create Ansible Inventory with Terraform?
• 👩 How to run an Ansible playbook using GitHub Action?

Local Experimentation & Advanced Topics

These articles cover setting up local test environments and exploring specific components in more detail. They are excellent for understanding the individual parts before or alongside deploying the full cloud infrastructure.

• 🧰 1: Design and Test Ansible playbook with Vagrant
  • Example code .articles/1_vagrant_101
• 🧰 2 Experimenting on Docker Swarm with Vagrant and Ansible
  • Example code .articles/2_docker_swarm_101
• WIP: 🧰 3: Automate Infrastructure provisioning with Ansible and Github action (This might be more of an advanced topic now, focusing on CI/CD aspects beyond the basic make up)

Learn about the applications used in this setup

• WIP: ☸️ 1: What is Portainer and why you might need it. (Covered by Portainer Management UI)
• WIP: ☸️ 2: What is Prometheus and why you might need it.
• WIP: ☸️ 3: What is Caddy and why you might need it. (Covered by Caddy Web Server)

Deep Dives

Below, I've included some explainers on how I solved a specific problem when I encountered it. This should demistify some of the magic happening in this repo.

Ansible

• 🏃‍♂️ 1: Speedup Ansible Playbook Pull-Requests by only running affected roles
• 🏃‍♂️ 2: How to rotate Docker Secrets with Ansible
• 🏃‍♂️ 3: How to implement Pull-Request locking for Ansible

Ansible Collection

This repository is also available as an Ansible Collection on Ansible Galaxy, allowing you to easily reuse the roles in your own Ansible projects.

Collection Name: xnok.infra_bootstrap_tools

Installation

To install this collection from Ansible Galaxy, use the following command:

ansible-galaxy collection install xnok.infra_bootstrap_tools

Usage

Once installed, you can use the roles from this collection in your playbooks. For example, to use the docker role:

- hosts: all
  become: yes
  roles:
    - role: xnok.infra_bootstrap_tools.docker
      # Optional: specify variables for the role
      # docker_users:
      #   - your_username

Refer to the README.md file within each role's directory (ansible/roles/[role_name]/README.md) for detailed information on specific roles, their variables, and dependencies.

You can find the collection on Ansible Galaxy: xnok.infra_bootstrap_tools

Architecture

Handy toolchain

Do you want to go fast? Too lazy to set up your local environment?

Then, use the tools from a Docker container. I included a simple Toochain in this repository and a useful alias for it.

Use common infrastructure tools in docker with:

• docker_tools_alias.sh

source ./bin/docker_tools_alias.sh

use dasb for ansible in docker
use dap for ansible-playbook in docker
use daws for awscli in docker
use dpk for packer in docker
use dtf for terraform in docker
use dbash for bash in docker

Tools Showcase

This project leverages several tools to streamline development and improve code quality. Here's a brief overview of some of them:

• Pre-commit: We use pre-commit hooks to automate linting, formatting, and other checks before code is committed. This helps maintain code consistency and catch errors early. Configuration can be found in .pre-commit-config.yaml.
• 1Password CLI: For securely managing sensitive information like API keys and passwords, the 1Password CLI can be integrated into your workflow. The setup script provides an option to install it.
• Boilerplate: Gruntwork Boilerplate is used to generate répétitive code structures, ensuring consistency and saving time. You can find boilerplate templates in the .boilerplates directory.
• Hugo: The project documentation website (what you're likely reading if you're on the website!) is built using Hugo, a fast and flexible static site generator. The website content is in the website/ directory.

Scale Up

With docker swarm and portainer it because easy to manager multiple nodes.