Flux2 Infra repository

This repository provides a collection of infrastructure applications and add-ons intended for deployment within Kubernetes clusters using FluxCD.

Note: This repository serves as a boilerplate and reference implementation. All configurations should be reviewed, adapted, and validated based on your specific requirements and environment. Do not deploy this setup directly into production without proper customization and testing.

Table of Contents

• Support Project
• Prerequisites
• List of applications
• Import current repository
  • Substitute variables
• Repository structure
• Structure idea
• Flex/Stable Bundles and promotion
• Slack Notifications
• Star History

Support Project

Prerequisites

• Kubernetes cluster version 1.24 or newer
• Flux version 2.3.0 or newer bootstrapped to the Head repository (example)
• CRD GitOps repository must be included before this as a dependency.

List of applications

Core Components

• aws-load-balancer-controller - Manages AWS Elastic Load Balancers for Kubernetes clusters
• cert-manager - X.509 certificate controller for Kubernetes
• cluster-autoscaler - a component that automatically adjusts the size of a Kubernetes Cluster
• external-dns - synchronizes exposed Kubernetes Services and Ingresses with DNS providers.
• external-secrets - External Secrets Operator (ESO) is used to synchronize secrets from external APIs into Kubernetes
• Istio ServiceMesh - ServiceMesh based on Envoy

Observability Components

• Victoria Metrics (victoria-metrics-k8s-stack) - Metrics Database
• Victoria Logs Single - Logs Database
• flux-monitoring - alerts and dashboards for the FluxCD
• Loki - Logs Database
• Grafana Alloy - Telemetry Collector
• blackbox-exporter - allows blackbox probing of endpoints
• helm-exporter - Exports Helm release, chart, and version metrics in Prometheus format.
• k8s-event-logger - This tool simply watches Kubernetes Events and logs them to stdout in JSON to be collected and stored by your logging solution
• kubelinks - Provides a web page with links to all URLs from Kubernetes ingresses and Istio gateways.
• metrics-server - collects resource metrics from Kubelets and exposes them in Kubernetes apiserver
• oomkill-exporter - Exports metrics about Out-Of-Memory (OOM) events in Kubernetes
• x509-certificate-exporter - Exports metrics about x509 certificate expiration and validity.

Security

• Kyverno - a cloud native policy engine

Nice to have

• reflector
• KEDA
• stakater/Reloader

Extra Components

• Apache Airflow
• Apache Superset - is an open-source modern data exploration and visualization platform.
• ArgoCD
• capacitor (Flux UI)
• Clickhouse Operator (Altinity)
• Clickhouse simple example (Altinity)
• cloudnative-pg (PostgreSQL operator)
• cloudnative-pg tenant example
• Dragonfly Instance (redis replacement)
• Dragonfly Operator (redis replacement)
• elastic operator (ECK)
• httpbin
• Jenkins
• kro (Kube Resource Orchestrator)
• minio-operator
• minio-tenant example
• n8n - AI workflow automation
• ollama & open-webui
• pgadmin
• SeaweedFS (S3 alternative)
• Strimzi Kafka Operator
• Valkey Instance - example of Valkey instance
• Valkey Operator - open-source for of redis by AWS

Import current repository

Create GitRepository and Flux Kustomization in your main repository (change path and substitute variables as needed):

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: fluxcd-gitops-infra
  namespace: flux-system
spec:
  interval: 10m
  ref:
    branch: main
  secretRef:
    name: flux-system
  url: https://github.com/brainfair/awesome-flux-infra.git
  ignore: |
 # exclude README.md
 /README.md
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: fluxcd-gitops-infra
  namespace: flux-system
spec:
  dependsOn:
 - name: fluxcd-gitops-crds
  interval: 10m
  path: ./clusters/homelab
  prune: true
  sourceRef:
    kind: GitRepository
    name: fluxcd-gitops-infra
  postBuild:
    substitute:
      env: "homelab"
      cluster_name: "docker-desktop"
      cluster_subdomain: "localhost.direct"

Substitute variables

• env - environment name
• cluster_name - the name of the Kubernetes cluster
• cluster_subdomain - subdomain for all ingress resources

Check head repository example

Repository structure

The Git repository contains the following top directories:

• apps/base dir contains base application objects such as helmrelease, helmrepository, namespace, etc...
• apps/bundles dir contains bundles for different types of environment
• clusters dir contains cluster entry point where we can include different bundles or custom apps

── apps
 ├── base
 ├   ├── aws-load-balancer-controller
 ├   ├── blackbox-exporter
 ├   ....
 ├── bundles
    ├── eks-flex
    ├── eks-stable
    ├── aks-flex
    ├── aks-stable
    ├── esxi-flex
    ├── esxi-stable
    ├── docker-flex
    ├── docker-stable
    ├── ...
── clusters
    ├── eks-cluster
    ├── aks-cluster
    ├── dockerdesktop-cluster
    ├── vmware-cluster
    ├── ...

Structure idea

The basic idea to define 3 levels of application kustomization:

• base level inside apps/base defines a common definition of application and values for everything
• bundles level inside apps/bundles defines aggregation of the base application and kustomization for the specified bundle
• cluster level inside clusters/[cluster_name] defines entry point where we can include some bundle and override cluster-specific values or add something more

Flex/Stable Bundles and promotion

To keep infrastructure up-to-date we defined two bundles for the Docker Desktop environment.

• docker-flex - where we define the version range for every application (blocking major updates) to allow automatic updates while maintaining stability. Switched to renovatebot to improve visibility and control of updates.
• docker-stable - where we define a pinned version for each application.

Non-production environment should include a flex bundle where we can play/test/evaluate new applications and new versions. Production environments should be a pointer to a stable bundle.

For 3rd party applications when a new version is successfully updated in the flex bundle we run the promotion workflow triggered by dispatch notification defined in the staging docker cluster.

Star History