ilert-kube-agent is a service that listens to the Kubernetes API server and generates alerts about the health state of the pods and the nodes.
The iLert kube agent requires an API key to send alerts to iLert. You can configure a single API key or multiple API keys separated by commas.
Single API Key:
settings:
apiKey: "your-single-api-key-here"
Multiple API Keys:
settings:
apiKey: "api-key-1,api-key-2,api-key-3"
When multiple API keys are provided, the agent will send alerts to all configured iLert alert sources. This is useful for:
- Sending alerts to multiple teams or environments
- Redundancy and failover scenarios
- Different alert routing based on API key configuration
Environment Variable:
You can also set the API key using the ILERT_API_KEY
environment variable:
export ILERT_API_KEY="api-key-1,api-key-2,api-key-3"
Simply build and run ilert-kube-agent to get Kubernetes cluster alarms.
Flag | Description |
---|---|
--alarms.cluster.enabled |
Enables cluster alarms. Triggers an alarm if any cluster problem occurred e.g. API server not available [Default: true] |
--alarms.pods.terminate.enabled |
Enables terminate pod alarms. Triggers an alarm if any pod terminated e.g. Terminated, OOMKilled, Error, ContainerCannotRun, DeadlineExceeded, Evicted [Default: true] |
--alarms.pods.waiting.enabled |
Enables waiting pod alarms. Triggers an alarm if any pod in waiting status e.g. CrashLoopBackOff, ErrImagePull, ImagePullBackOff, CreateContainerConfigError, InvalidImageName, CreateContainerError [Default: true] |
--alarms.pods.restarts.enabled |
Enables restarts pod alarms. Triggers an alarm if any pod restarts count reached threshold [Default: true] |
--alarms.pods.resources.cpu.enabled |
Enables pod CPU resource alarms. Triggers an alarm if any pod reaches CPU limit [Default: true] |
--alarms.pods.resources.memory.enabled |
Enables pod CPU resource alarms. Triggers an alarm if any pod reaches memory limit [Default: true] |
--alarms.nodes.terminate.enabled |
Enables terminate node alarms. Triggers an alarm if any node terminated. [Default: true] |
--alarms.nodes.resources.cpu.enabled |
Enables node CPU resource alarms. Triggers an alarm if any node reaches CPU limit [Default: true] |
--alarms.nodes.resources.memory.enabled |
Enables node CPU resource alarms. Triggers an alarm if any node reaches memory limit [Default: true] |
Note: Google Kubernetes Engine (GKE) Users - GKE has strict role permissions that will prevent the kube-state-metrics roles and role bindings from being created. To work around this, you can give your GCP identity the cluster-admin role by running the following one-liner:
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud info --format='value(config.account)')
helm repo add ilert https://ilert.github.io/charts/
helm repo update
helm upgrade --install --namespace kube-systems \
ilert-kube-agent ilert/ilert-kube-agent \
--set config.settings.apiKey="<YOUR KEY HERE>" \
--set inClusterRoutesEnabled=true \
--set config.settings.httpAuthorizationKey="<HTTP AUTHORIZATION HERE>"
Note: In-cluster HTTP routes are only functional when both inClusterRoutesEnabled
is true
and config.settings.httpAuthorizationKey
is set. If config.settings.httpAuthorizationKey
is not given, a random alphanumeric string with 64 characters will be generated.
Multiple API Keys: To use multiple API keys, separate them with commas:
helm upgrade --install --namespace kube-systems \
ilert-kube-agent ilert/ilert-kube-agent \
--set config.settings.apiKey="key1,key2,key3" \
--set inClusterRoutesEnabled=true \
--set config.settings.httpAuthorizationKey="<HTTP AUTHORIZATION HERE>"
- Define module:
module "ilert-kube-agent" {
source = "iLert/ilert-kube-agent/kubernetes"
replicas = 2
api_key = "<YOUR KEY HERE>"
http_authorization_key = "<HTTP AUTHORIZATION HERE>"
}
Multiple API Keys: To use multiple API keys, separate them with commas:
module "ilert-kube-agent" {
source = "iLert/ilert-kube-agent/kubernetes"
replicas = 2
api_key = "key1,key2,key3"
http_authorization_key = "<HTTP AUTHORIZATION HERE>"
}
- Apply changes
terraform init
terraform apply
Note: In-cluster HTTP routes are only functional when the http_authorization_key
is set.
Follow this guide please.
To deploy this project, you can simply run kubectl apply -f deployment/standard
and a
Kubernetes service and deployment will be created.
We are happy to respond to GitHub issues as well.
Licensed under Apache License, Version 2.0
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in ilert-kube-agent by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.