📊 Grafana + Teleport Metrics Integration (Demo Project)

[pnrao1983]

📊 Grafana + Teleport Metrics Integration (Demo Project)

This is a complete show and tell for deploying Grafana with Teleport metrics on an EC2 instance using Terraform and Amazon Linux, exposing diagnostic metrics, and creating a rich dashboard to monitor key Teleport metrics.

🚀 Demo Goals

Deploy a Grafana instance on Amazon Linux EC2 using Terraform

Expose Teleport metrics via diag_addr

Set up Prometheus to scrape these metrics

Import a prebuilt Grafana dashboard (Your-cluster Tenant Details)

🧱 Infrastructure Setup

✅ Terraform Prerequisites

VPC ID: vpc-[your_VPC_details]

Subnet ID: subnet-[your_subnet_details]

Security Group: sg-[your_security_group_details]

Keypair: [your_key_name]

Amazon Linux 3 AMI (region: us-west-1): ami-[your_AMI_details]

🧾 Terraform Code Snippet

resource "aws_instance" "grafana" {
ami = "ami-[your_AMI_details]"
instance_type = "t2.medium"
key_name = "[your_key_name]"
subnet_id = "subnet-[your_subnet_details]"
vpc_security_group_ids = "sg-[your_security_group_details]"
associate_public_ip_address = true

tags = {
Name = "GrafanaServer"
instance_metadata_tagging_req = "user@domain.com"
}

user_data = <<-EOF
#!/bin/bash
sudo yum update -y
sudo amazon-linux-extras enable epel
sudo yum install -y wget

sudo tee /etc/yum.repos.d/grafana.repo <<EOL
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
EOL

sudo yum install -y grafana
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

EOF
}

📡 Teleport Metric Exposure

Update /etc/teleport.yaml:

teleport:
diag_addr: 127.0.0.1:3000

This exposes metrics at http://127.0.0.1:3000/metrics

Test locally:

curl http://127.0.0.1:3000/metrics | grep watcher_events_count

📈 Prometheus Setup (Binary Based)

wget https://github.com/prometheus/prometheus/releases/download/v3.3.0-rc.0/prometheus-3.3.0-rc.0.linux-amd64.tar.gz
cd prometheus-3.3.0-rc.0.linux-amd64
./prometheus --config.file=prometheus.yml

prometheus.yml:

scrape_configs:

• job_name: "teleport"
  static_configs:
  • targets: ["localhost:3000"]

📦 Importing the Dashboard

Upload or provision the following file:

Nara-cluster Tenant Details-1744898831067.json

Manual Import Steps:

Go to Grafana → + Create → Import

Upload JSON or paste content

Select data source (e.g., Prometheus)

Click Import
Below is a sample screenshot of the Grafana Dashboard look like:

📊 Sample Metrics Covered

user_max_concurrent_sessions_hit_total

teleport_kubernetes_server_exec_in_flight_sessions

dynamo_requests_total

user_login_total

teleport_roles_total

watcher_events_count

🔒 Gotchas & Fixes

Ensure security group allows inbound 3000 and 9090 ports

Use correct Prometheus scrape intervals

Make sure grafana.ini has proper permissions if running systemd

Teleport must be configured with diag_addr for metrics

📥 Example API Import Command

curl -X POST http://<GRAFANA_HOST>:3000/api/dashboards/db
-H "Content-Type: application/json"
-H "Authorization: Bearer <GRAFANA_API_KEY>"
-d @"Nara-cluster Tenant Details-1744898831067.json"