ilert-kube-agent

ilert-kube-agent is a service that listens to the Kubernetes API server and generates alerts about the health state of the pods and the nodes.

_{Node alert example:}

_{Pod alert example:}

Configuration

API Key Configuration

The iLert kube agent requires an API key to send alerts to iLert. You can configure a single API key or multiple API keys separated by commas.

Single API Key:

settings:
  apiKey: "your-single-api-key-here"

Multiple API Keys:

settings:
  apiKey: "api-key-1,api-key-2,api-key-3"

When multiple API keys are provided, the agent will send alerts to all configured iLert alert sources. This is useful for:

• Sending alerts to multiple teams or environments
• Redundancy and failover scenarios
• Different alert routing based on API key configuration

Environment Variable: You can also set the API key using the ILERT_API_KEY environment variable:

export ILERT_API_KEY="api-key-1,api-key-2,api-key-3"

Usage

Simply build and run ilert-kube-agent to get Kubernetes cluster alarms.

Flag	Description
--alarms.cluster.enabled	Enables cluster alarms. Triggers an alarm if any cluster problem occurred e.g. API server not available [Default: true]
--alarms.pods.terminate.enabled	Enables terminate pod alarms. Triggers an alarm if any pod terminated e.g. Terminated, OOMKilled, Error, ContainerCannotRun, DeadlineExceeded, Evicted [Default: true]
--alarms.pods.waiting.enabled	Enables waiting pod alarms. Triggers an alarm if any pod in waiting status e.g. CrashLoopBackOff, ErrImagePull, ImagePullBackOff, CreateContainerConfigError, InvalidImageName, CreateContainerError [Default: true]
--alarms.pods.restarts.enabled	Enables restarts pod alarms. Triggers an alarm if any pod restarts count reached threshold [Default: true]
--alarms.pods.resources.cpu.enabled	Enables pod CPU resource alarms. Triggers an alarm if any pod reaches CPU limit [Default: true]
--alarms.pods.resources.memory.enabled	Enables pod CPU resource alarms. Triggers an alarm if any pod reaches memory limit [Default: true]
--alarms.nodes.terminate.enabled	Enables terminate node alarms. Triggers an alarm if any node terminated. [Default: true]
--alarms.nodes.resources.cpu.enabled	Enables node CPU resource alarms. Triggers an alarm if any node reaches CPU limit [Default: true]
--alarms.nodes.resources.memory.enabled	Enables node CPU resource alarms. Triggers an alarm if any node reaches memory limit [Default: true]

Deployment

Note: Google Kubernetes Engine (GKE) Users - GKE has strict role permissions that will prevent the kube-state-metrics roles and role bindings from being created. To work around this, you can give your GCP identity the cluster-admin role by running the following one-liner:

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud info --format='value(config.account)')

Helm Deployment (recommended)

helm repo add ilert https://ilert.github.io/charts/
helm repo update

helm upgrade --install --namespace kube-systems \
    ilert-kube-agent ilert/ilert-kube-agent \
    --set config.settings.apiKey="<YOUR KEY HERE>" \
    --set inClusterRoutesEnabled=true \
    --set config.settings.httpAuthorizationKey="<HTTP AUTHORIZATION HERE>"

Note: In-cluster HTTP routes are only functional when both inClusterRoutesEnabled is true and config.settings.httpAuthorizationKey is set. If config.settings.httpAuthorizationKey is not given, a random alphanumeric string with 64 characters will be generated.

Multiple API Keys: To use multiple API keys, separate them with commas:

helm upgrade --install --namespace kube-systems \
    ilert-kube-agent ilert/ilert-kube-agent \
    --set config.settings.apiKey="key1,key2,key3" \
    --set inClusterRoutesEnabled=true \
    --set config.settings.httpAuthorizationKey="<HTTP AUTHORIZATION HERE>"

Terraform Deployment (recommended)

• Define module:

module "ilert-kube-agent" {
  source  = "iLert/ilert-kube-agent/kubernetes"
  replicas = 2
  api_key = "<YOUR KEY HERE>"
  http_authorization_key = "<HTTP AUTHORIZATION HERE>"
}

Multiple API Keys: To use multiple API keys, separate them with commas:

module "ilert-kube-agent" {
  source  = "iLert/ilert-kube-agent/kubernetes"
  replicas = 2
  api_key = "key1,key2,key3"
  http_authorization_key = "<HTTP AUTHORIZATION HERE>"
}

• Apply changes

terraform init
terraform apply

Note: In-cluster HTTP routes are only functional when the http_authorization_key is set.

Lambda Deployment

Follow this guide please.

Raw YAML Deployment

To deploy this project, you can simply run kubectl apply -f deployment/standard and a Kubernetes service and deployment will be created.

Getting help

We are happy to respond to GitHub issues as well.

License

^{Licensed under Apache License, Version 2.0}
_{Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in ilert-kube-agent by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.}