Heartbeats

---

A lightweight HTTP service for monitoring periodic "heartbeat" pings ("bumps") and notifying configured receivers when a heartbeat goes missing or recovers. Includes an in‐browser read-only dashboard showing current heartbeats, receivers, and historical events.

🚀 Features

• Heartbeat monitoring with configurable interval & grace periods
• Pluggable notifications via Slack, Email, or MS Teams
• In-memory history of recent events (received, failed, state changes, notifications, API requests)
  • Stores up to --history-size entries (default: 10,000)
  • Not persisted across restarts (~1.73 MB at max size)
• Dashboard with:
  • Heartbeats: status, URL, last bump, receivers, quick-links
  • Receivers: type, destination, last sent, status
  • History: timestamped events, filter by heartbeat
  • Text-search filters & copy-to-clipboard URLs
• /healthz and /metrics endpoints for health checks & Prometheus
• YAML configuration with variable resolution via containeroo/resolver

🏁 Flags

Flag	Shorthand	Default	Environment Variable	Description
--config	-c	heartbeats.yml	HEARTBEATS_CONFIG	Path to configuration file
--listen-address	-a	:8080	HEARTBEATS_LISTEN_ADDRESS	Address to listen on (host:port)
--route-prefix		``	HEARTBEATS_ROUTE_PREFIX	ath prefix to mount the app (e.g., /heartbeats).
--site-root	-r	http://localhost:8080	HEARTBEATS_SITE_ROOT	Base URL for dashboard and link rendering
--history-size	-s	10000	HEARTBEATS_HISTORY_SIZE	Number of historical events to keep in memory (default ~1.73 MB for 10,000). Not persisted.
--skip-tls	-	false	HEARTBEATS_SKIP_TLS	Skip TLS verification for all receivers (can be overridden per receiver)
--debug	-d	false	HEARTBEATS_DEBUG	Enable debug-level logging
--debug-server-port	-p	8081	HEARTBEATS_DEBUG_SERVER_PORT	Port for the debug server
--log-format	-l	text	HEARTBEATS_LOG_FORMAT	Log format (json or text)
--retry-count	-	3	HEARTBEATS_RETRY_COUNT	Number of times to retry a failed notification. Use -1 for infinite.
--retry-delay	-	2s	HEARTBEATS_RETRY_DELAY	Delay between retries. Must be ≥ 1s.
--help	-h	-	-	Show help and exit
--version	-	-	-	Print version and exit

Proxy Environment Variables

• HTTP_PROXY: URL of the proxy server to use for HTTP requests
• HTTPS_PROXY: URL of the proxy server to use for HTTPS requests

🌐 Endpoints

Path	Method	Description
/	GET	Dashboard home page
/bump/{id}	POST, GET	Create a new heartbeat
/bump/{id}/fail	POST, GET	Manually mark heartbeat as failed
/healthz	GET	Liveness probe
/metrics	GET	Prometheus metrics endpoint

⚙️ Configuration

heartbeats and receivers must be defined in your YAML file (default config.yaml).

Examples (config.yaml)

---
receivers:
  dev-crew-int:
    slack_configs:
      - channel: integration
        token: env:SLACK_TOKEN
heartbeats:
  prometheus-int:
    description: "Prometheus → Alertmanager test"
    interval: 15m
    grace: 90s
    receivers:
      - dev-crew-int

Heartbeats

A heartbeat waits for periodic pings ("bumps"). If no bump arrives within interval + grace, notifications are sent.

To reduce noise from race conditions (e.g. pings arriving milliseconds after grace timeout), Heartbeats adds a short internal delay before transitioning to grace or missing. This ensures smoother handling of near-expiry bumps without affecting responsiveness.

Key	Type	Description
description	string	(optional) Human-friendly description
interval	duration	Required. Go duration (e.g. 15m, 90s) for expected interval between pings
grace	duration	Required. Go duration after interval before marking missing
receivers	[]string	Required. List of receiver IDs (keys under receivers:) to notify upon missing

Example

heartbeats:
  prometheus-int:
    description: "Prometheus → Alertmanager test"
    interval: 15m
    grace: 90s
    receivers:
      - dev-crew-int

Receivers

Each receiver can have multiple notifier configurations. Supported under receivers::

• slack_configs
• email_configs
• msteams_configs

You may use any template variable from the heartbeat (e.g. {{ .ID }}, {{ .Status }}), and these helper functions:

• upper: {{ upper .ID }}
• lower: {{ lower .ID }}
• formatTime: {{ formatTime .LastBump "2006-01-02 15:04:05" }}
• ago: {{ ago .LastBump }}
• isRecent: {{ isRecent .LastBump }} // isRecent returns true if the last bump was less than 2 seconds ago
• join: {{ join .Tags ", " }}

Variable Resolution

Heartbeats uses containeroo/resolver for variable resolving.

Resolver supports:

• Plain: literal value
• Environment: env:VAR_NAME
• File: file:/path/to/file
• Within-file: file:/path/to/file//KEY, also supported yaml:,json:,ini: and toml:. For more details see containeroo/resolver.

Slack

Defaults:

• subject_tmpl: [{{ upper .Status }}] {{ .ID }}"
• text_tmpl: {{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"

receivers:
  dev-crew-int:
    slack_configs:
      - channel: "#integration"
        token: env:SLACK_TOKEN
        # optional custom templates:
        title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
        text_tmpl: "{{ .ID }} status: {{ .Status }}"
        # optional: override global skip TLS
        skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

Email

Defaults:

• subject_tmpl: "[HEARTBEATS]: {{ .ID }} {{ upper .Status }}"
• body_tmpl: "<b>Description:</b> {{ .Description }}<br>Last bump: {{ ago .LastBump }}"

email_configs:
  - smtp:
      host: smtp.gmail.com
      port: 587
      from: admin@example.com
      username: env:EMAIL_USER
      password: env:EMAIL_PASS
      # optional
      start_tls: true
      # optional: override global skip TLS
      skip_insecure_verify: true
    email:
      is_html: true
      to: ["ops@example.com"]
      # optional custom templates:
      subject_tmpl: "[HB] {{ .ID }} {{ upper .Status }}"
      body_tmpl: "Last bump: {{ ago .LastBump }}"

MS Teams (incomming webhook)

Defaults:

• title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
• text_tmpl: "{{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"

msteams_configs:
  - webhook_url: file:/secrets/teams/webhook//prod
    # optional custom templates:
    title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
    text_tmpl: "{{ .ID }} status: {{ .Status }}"
    # optional: override global skip TLS
    skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

MS Teams (Graph API) (NOT TESTED)

Defaults:

• title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
• text_tmpl: "{{ .ID }} is {{ .Status }} (last bump: {{ ago .LastBump }})"

msteamsgraph_configs:
  - channel_id: env:MSTEAMSGRAPH_CHANNEL_ID
    team_id: env:MSTEAMSGRAPH_TEAM_ID
    # optional custom templates:
    title_tmpl: "[{{ upper .Status }}] {{ .ID }}"
    text_tmpl: "{{ .ID }} status: {{ .Status }}"
    # optional: override global skip TLS
    skip_tls: true

Heartbeats adds a custom User-Agent: Heartbeats/<version> header to all outbound HTTP requests. The Content-Type header is also set to application/json.

📈 Metrics

Heartbeats exposes the following Prometheus metrics via the /metrics endpoint:

Heartbeat Metrics

Name	Type	Labels	Description
heartbeats_heartbeat_last_status	gauge	heartbeat	Most recent status of each heartbeat (0 = DOWN, 1 = UP)
heartbeats_heartbeat_received_total	counter	heartbeat	Total number of received heartbeats per ID
heartbeats_receiver_last_status	counter	receiver, type, target	Reports the status of the last notification attempt (1 = ERROR, 0 = SUCCESS)

History Store Metrics

Name	Type	Description
heartbeats_history_byte_size	gauge	Current size of the in-memory history store in bytes

🧪 Development

Download the binary and update the example config.yaml according your needs. If you prefer to run heartbeats in docker, you find a docker-compose.yaml & config.yaml here. For a kubernetes deployment you find the manifests here.

🔧 Debugging

Heartbeats includes optional internal endpoints for testing receiver notifications and simulating heartbeats. These are only enabled when the --debug flag is set.

Internal Endpoints

Path	Method	Description
/internal/receiver/{id}	GET	Sends a test notification to the receiver
/internal/heartbeat/{id}	GET	Simulates a bump for the given heartbeat

These endpoints listen only on 127.0.0.1.

Kubernetes

Forward the debug port to your local machine:

kubectl port-forward deploy/heartbeats 8081:8081

curl http://localhost:8081/internal/receiver/{id}
curl http://localhost:8081/internal/heartbeat/{id}

Docker

Bind the debug port only to your host's loopback interface:

docker run \
  -p 127.0.0.1:8081:8081 \
  -v ./config.yaml:/config.yaml \
  containeroo/heartbeats

curl http://localhost:8081/internal/receiver/{id}
curl http://localhost:8081/internal/heartbeat/{id}

✅ This ensures /internal/* endpoints are only reachable from your local machine, not from other containers or the network.

⚠️ Warning: These endpoints are meant for local testing and debugging only. Never expose them in production.

📝 License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.