High Availability Setup

[hhftechnology]

for critical deployments
deploying multiple Gerbil instances:

services:
  gerbil-1:
    image: fosrl/gerbil:1.0.0-beta.3
    command:
      - --reachableAt=http://gerbil-1:3003
      - --subnet_group=100.89.137.0/20
  gerbil-2:
    image: fosrl/gerbil:1.0.0-beta.3
    command:
      - --reachableAt=http://gerbil-2:3003
      - --subnet_group=100.89.138.0/20

Since Gerbil acts as the WireGuard endpoint, we need to ensure WireGuard clients can seamlessly switch to a backup endpoint if the primary fails.

implement DNS-based failover
For DNS-based failover of WireGuard endpoints:

1. Configure multiple A records for your WireGuard endpoint domain that each point to different Gerbil instances:

# Example DNS configuration
wg.example.com IN A 203.0.113.1  # Primary Gerbil
wg.example.com IN A 203.0.113.2  # Secondary Gerbil

2. Update the Pangolin configuration to use this DNS name:

gerbil:
    base_endpoint: wg.example.com

3. Configure your DNS provider's health checks and failover policy. Most providers like Cloudflare or AWS Route53 support this:

# Example Route53 health check configuration
- Name: "Gerbil Health Check"
  Type: "TCP"
  Port: 51820
  ResourcePath: "/"
  FailureThreshold: 3
  RequestInterval: 30

[hhftechnology]

For SQLite replication with Pangolin, we can use a file-based replication approach:

Set up a file sync tool like Litestream or rqlite that's specifically designed for SQLite replication.
I have this as a setup and working fine.

services:
  pangolin:
    # ... existing pangolin config ...
    volumes:
      - ./config:/app/config  # Contains SQLite database

  litestream:
    image: litestream/litestream
    volumes:
      - ./config:/data  # Mount same volume as Pangolin
    command: replicate /data/db/db.sqlite /path/to/replica/db.sqlite

[hhftechnology]

In theory to setup HAProxy to load balance the Pangolin and Gerbil services themselves for high availability:

services:
  haproxy:
    image: haproxy:2.4
    ports:
      - "80:80"
      - "443:443"
      - "51820:51820/udp"  # WireGuard port
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
      - ./certs:/etc/ssl/private:ro  # For SSL termination if needed
    depends_on:
      - pangolin-1
      - pangolin-2
      - gerbil-1 
      - gerbil-2

# haproxy.cfg

global
    log stdout format raw local0
    maxconn 4096

defaults
    log     global
    mode    http
    option  httplog
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/ssl/private/cert.pem
    
    # ACL for Pangolin web interface/API (this i am not sure of)
    acl is_pangolin path_beg /api/v1
    acl is_pangolin path_beg /auth
    acl is_pangolin path_beg /dashboard
    
    use_backend pangolin_servers if is_pangolin
    default_backend gerbil_servers

backend pangolin_servers
    balance roundrobin
    option httpchk GET /api/v1/health
    server pangolin1 pangolin-1:3000 check
    server pangolin2 pangolin-2:3000 check backup

backend gerbil_servers
    balance roundrobin
    option httpchk GET /health
    server gerbil1 gerbil-1:3003 check
    server gerbil2 gerbil-2:3003 check backup

frontend wireguard_front
    bind *:51820 udp
    mode tcp
    default_backend wireguard_servers

backend wireguard_servers
    mode tcp
    balance roundrobin
    server gerbil1 gerbil-1:51820 check
    server gerbil2 gerbil-2:51820 check backup

In theory

1. Load balances HTTP/HTTPS traffic between Pangolin instances
2. Load balances WireGuard traffic between Gerbil instances
3. Performs health checks on both services
4. Uses backup servers that only receive traffic if primary fails
5. Handles both web traffic and WireGuard UDP traffic (very imp)

[MK796]

But haproxy would then be a single point of failure... but cool suggestion