This project is focused on using Prometheus for monitoring on different levels and also implementing Grafana for data visualization :
- Infrastructure Level: CPU, RAM, Network etc
- Platform Level: Kubernetes components
- Application Level: 3rd-party and Own application
Requirements:
- Ensure AWS-CLI, Kubectl and Helm are installed
- Configure AWS credentials on local machine with admin access (for project purpose)
Setup EKS cluster using EKSCTL command.
Connect to the cluster, add prometheus chart and install prometheus chart for kubernetes into the cluster
Port forward to grafana and prometheus service to connect to the U.I
Access UI for prometheus and grafana (enter required credentials)
List below are the metrics gotten from the clusters including the nodes also
Below is a detailed metrics for the cluster
Deployed a microservice application
We will launch a container dedicated to using the Curl command on the load balancer. This involves connecting to the applications through a script that establishes a loop, which create a high traffic
The expression rate(node_cpu_seconds_total{mode="idle"}[2m]) is a Prometheus query that calculates the per-second rate of change for the node_cpu_seconds_total metric with the mode label set to `"idle". This type of expression is what is used by grafana to display metrics needed.
There is need to get notified when something goes wrong, this is where alerting comes into play
- Define what we want to be notified about e.g when CPU usage is above is 50%
- Send notification i.e email/slack notification (will be implemented later)
We will set alert rules on any usecase we are interested e.g pod not starting. The prometheusrule handles the rules
When setting up this rules, its imperative to make use of the appropriate labels in the rules we plans on setting up, the highlighted lables below needs to be specified in our rule.
We are setting two alerts respectively downand HostHighCpuLoad. The labels are included so that prometheusrule can include it in the alert that it has to monitor
When you apply this rules, ensure you check prometheus log of config-reloader container to confirm a reload has triggered(which implies the rule has been added) by checking the time and confirm it occured after applying the rule.
The alerts have been updated.
Let run cpustress container to generate CPU load and see if the alert will be triggered.
We just completed Kubernetes-level monitoring. Now, we are turning our attention to monitoring Redis, a third-party application within our microservices. To achieve this, we are utilizing an exporter that fetches metric data from the service and transforms the service-specific metrics into a format understandable by Prometheus. This exporter exposes the translated metrics through the /metrics endpoint. To ensure Prometheus is aware of this new exporter, we will deploy a ServiceMonitor, a custom Kubernetes resource designed for this purpose.
Deploy redis exporter, ensure you set servicemonitor to true and specify the redis service name in the helms value
Redis-exporter is deployed by checking the targets under the status section and we can query different redis metrics.
We can see the redis-exporter pod
We can setup a prometheusrule using ready made rules Exporter rules to define the rules to specify for our exporters. For our redis application, we will be using Redis rule, like the screenshot below.
Ensure you specified the appropriate labels and apply the rule
Let edit the redis pod replica to zero, to see if the redis-exporter will be triggered.
Now lets try and create a dashboard for our redis application, we can make use of ready made dashboards from Grafana, we will be making use on one of the numerous dashoard for Redis
Copy the ID number and import the dashboard in grafana using the number, ensure you select data source as prometheus.
There are other redis dashboards you can also use. The one below is with ID 15895.
Lastly, we want to monitor our own application deployed in the cluster and we are using a nodejs application. There are no exporter available for our own application, so we have to define the metrics (custom solution). There are prometheus client library for different languages in which the application is written it. Developers write the code using the prometheus client library. The screenshot shows metrics to monitor in the application
We need to ensure to setup a servicemonitor with the appropriate label for the application nodeappand prometheus release=prometheus respectively so that the targets can be included in prometheus for monitoring
The targets have been included and we can query metrics for the application
Define the metrics we want to monitor and setup a grafana dashboard to display this metrics.
Ensure you delete the project when finished
When trying to delete the cluster you might get the error below
Follow the command below to resolve
