Homelab: Kubernetes Home Cluster - Applications

This repository contains applications deployed on the home-cluster via Flux using GitOps.

---

Bootstrapping

A Kubernetes cluster needs to be bootstrapped with the Cilium CNI and Flux pointing to this repository.

For ksops and Flux to decrypt the initial secrets for configuring the External Secrets Operator using HashiCorp Vault, a Google Cloud Service Account with access to the correct KMS key needs to be set in the flux namespace.

Attention: some applications will be automatically deployed, others not (yet).

---

App-of-Apps

The repository follows the app-of-apps pattern.

The first Flux Kustomization being defined needs to reference app-of-apps/.

These are bootstrapping the main Flux applications, referring to the respective <PROJECT>/applications/ kosutomizations:

• infrastructure: core cluster infrastructure
• core: core applications
• applications: (user) applications running on the cluster/network
• home-assistant: Home Assistant related applications

Each of these applications follows the app-of-apps pattern again using sub-kustomizations defined in the respective application directories.

---

Hosted Services

Infrastructure

The following applications are defined in infrastructure/.

• Cilium - Provides the cluster CNI.
• External Secrets Operator - Synchronizes secrets from external stores to Kubernetes Secret objects.
  • External Secrets Stores - Deploys the required ClusterSecretStores and Vault credentials as Kubernetes Secrets.
• Kubelet Serving Cert Approver - Enables automatic certificate approval by the kubelet.
• MetalLB - Provides a Kubernetes network load balancer to expose Kubernetes Services.
• Metrics Server - Collects container resource metrics.
• NVIDIA Device Plugin - Makes the NVIDIA GPU accessible in the cluster.
• Reloader - Automatically reloads Kubernetes resources when secrets or configmaps change.
• Rook Ceph - Manages persistent storage in the cluster.
• Traefik - Exposes Kubernetes Ingress resources to the "outside world".

Core Applications

The following applications are defined in core/.

• Cert Manager - Certificate management using ACME Let's Encrypt.
• CloudNativePG - PostgreSQL database operator.
• External DNS with Google Cloud DNS integration - Creates DNS records in Google Cloud DNS domains for publicly reachable services.
• Gatus - Service status page.
• Victoria Metrics Stack - Monitoring stack using Victoria Metrics.
• Grafana - Visualization of metrics, and other data.
• Velero - Performs cluster backups.
  • Includes deployment of backup schedules.

(User) Applications

The following applications are defined in applications/.

• External DNS with Adguard integration - Creates DNS records in Adguard for internal, local only, reachable services. Uses the External DNS Adguard Webhook provider.
• External Services - Deploys Kubernetes Services and Ingresses to local endpoints, and existing services outside of the cluster.
• Immich - Photo management solution.
• InfluxDB - Time-Series database.
• LibreChat - Open-source chat application for AI conversations.
• Mealie - Recipe management application.
• Ollama - Run LLM models locally. (testing)
• Omada Controller - TP-Link Omada Controller.

Home Assistant

The following applications are defined in home-assistant/.

• ecowitt2mqtt - Forwards data received from ecowitt devices to the MQTT broker.
• EMQX - A MQTT broker.
• Home Assistant - The Home Assistant instance.
  • PostgreSQL instance as the Home Assistant recorder target and configured via the CloudNativePG operator.
• Node-RED - Automation based on flows and Home Assistant data.
• Ring MQTT - Amazon Ring devices to MQTT bridge.
• Telegraf - Forwards Home Assistant state changes to a local InfluxDB instance.
• Z-Wave JS - Full featured Z-Wave Control Panel and MQTT Gateway.
• Faster Whisper - Faster Whisper transcription with CTranslate2. (testing)
• OpenWakeWord - An open-source audio wake word (or phrase) detection framework. (testing)
• Piper - A local TTS server. (testing)

Notes: Backup and Restore

Home Assistant related backup and restore is handled via S3 backups.

The following services implement an initContainer as well as a nightly CronJob to backup data to an S3 bucket. If no data is found in the Persistent Volume yet, the data from will be retrieved and copied over which results in a full restore.

• Ring MQTT

The following services use API calls to determine whether a backup or restore is necessary.

• Node-RED
• Home Assistant
• Z-Wave JS UI

The following services also have Git repositories to store their configuration which gets pulled in upon start.

• Home Assistant
  • Home Assistant also defines it's own backup method via a trigger and a shell_command, and doesn't rely on a CronJob.
• Ring MQTT

---

Backup and Restore

The current backup and restore strategy consists of:

• CloudNativePG backups for persistent PostgreSQL data
• Home Assistant: see (#notes-backup-and-restore)
• Velero as a second layer disaster recovery for critical workloads

Timewise, the layers of backups follow the strategy:

1. 12:00am: in-application backups
2. 02:00am: Velero backups

---

Resource Optimization

Kubernetes Resource Recommendations can be used to analyze the resource usage of the cluster and provide recommendations for optimizing the resource requests and limits.

kubectl apply -f https://raw.githubusercontent.com/robusta-dev/krr/refs/heads/main/docs/krr-in-cluster/krr-in-cluster-job.yaml
kubectl logs -l batch.kubernetes.io/job-name=krr > krr.txt
kubectl delete -f https://raw.githubusercontent.com/robusta-dev/krr/refs/heads/main/docs/krr-in-cluster/krr-in-cluster-job.yaml

By default, this generates a text file with the recommendations. To output any other format, you can use the -f flag followed by the desired format.

If using JSON, you can use the jq command to get a list of all changes:

# get all current CPU requests
cat krr.json | jq '.scans[].object.allocations.requests.cpu | select(. != "?") | select(. != null)' | awk '{ sum += $0 } END { print sum }'
# get all recommended CPU requests
cat krr.json | jq '.scans[].recommended.requests.cpu | select(.value != "?") | .value' | awk '{ sum += $0 } END { print sum }'

# get all current memory requests
cat krr.json | jq '.scans[].object.allocations.requests.memory | select(. != "?") | select(. != null)' | awk '{ sum += $0 } END { print sum/1.074e9 }'
# get all current memory limits
cat krr.json | jq '.scans[].object.allocations.limits.memory | select(. != "?") | select(. != null)' | awk '{ sum += $0 } END { print sum/1.074e9 }'
# get all recommended memory requests (= limits)
cat krr.json | jq '.scans[].recommended.requests.memory | select(.value != "?") | .value' | awk '{ sum += $0 } END { print sum/1.074e9 }'

---

Continuous Integration and Automations

• GitHub Actions are linting all YAML files.
• Renovate Bot is updating Helm releases and used container images in the values.yaml files, and GitHub Actions.