|
| 1 | +# Single Cluster Open Source Observability - Istio Monitoring |
| 2 | + |
| 3 | +## Objective |
| 4 | + |
| 5 | +Service Meshes are an integral part of the Kubernetes environment that enables secure, reliable, and observable communication. Istio is an open-source service mesh that provides advanced network features without requiring any changes to the application code. These capabilities include service-to-service authentication, monitoring, and more. |
| 6 | + |
| 7 | +Istio generates detailed telemetry for all service communications within a mesh. This telemetry provides observability of service behavior, thereby empowering operators to troubleshoot, maintain, and optimize their applications. These features don’t impose additional burdens on service developers. To monitor service behavior, Istio generates metrics for all service traffic in, out, and within an Istio service mesh. These metrics provide information on behaviors, like traffic volume, traffic error rates, and request-response latency. |
| 8 | + |
| 9 | +In addition to monitoring the behavior of services within a mesh, it’s essential to monitor the behavior of the mesh itself. Istio components export metrics which provides insights into the health and function of the mesh control plane. |
| 10 | + |
| 11 | +This pattern configures an Amazon Elastic Kubernetes Service (Amazon EKS) cluster with Istio as a service mesh, Amazon Managed service for Prometheus, and Amazon Managed Grafana for monitoring your Istio Control and Data plane metrics |
| 12 | + |
| 13 | + |
| 14 | +## Prerequisites |
| 15 | + |
| 16 | +Ensure that you have installed the following tools on your machine. |
| 17 | + |
| 18 | +1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) |
| 19 | +2. [kubectl](https://Kubernetes.io/docs/tasks/tools/) |
| 20 | +3. [cdk](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install) |
| 21 | +4. [npm](https://docs.npmjs.com/cli/v8/commands/npm-install) |
| 22 | + |
| 23 | +## Deploying |
| 24 | + |
| 25 | +Please follow the _Deploying_ instructions of the [New EKS Cluster Open Source Observability Accelerator](./single-new-eks-opensource-observability.md) pattern, except for step 7, where you need to replace "context" in `~/.cdk.json` with the following: |
| 26 | + |
| 27 | +```typescript |
| 28 | + "context": { |
| 29 | + "fluxRepository": { |
| 30 | + "name": "grafana-dashboards", |
| 31 | + "namespace": "grafana-operator", |
| 32 | + "repository": { |
| 33 | + "repoUrl": "https://github.com/aws-observability/aws-observability-accelerator", |
| 34 | + "name": "grafana-dashboards", |
| 35 | + "targetRevision": "main", |
| 36 | + "path": "./artifacts/grafana-operator-manifests/eks/infrastructure" |
| 37 | + }, |
| 38 | + "values": { |
| 39 | + "GRAFANA_CLUSTER_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/cluster.json", |
| 40 | + "GRAFANA_KUBELET_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/kubelet.json", |
| 41 | + "GRAFANA_NSWRKLDS_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/namespace-workloads.json", |
| 42 | + "GRAFANA_NODEEXP_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/nodeexporter-nodes.json", |
| 43 | + "GRAFANA_NODES_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/nodes.json", |
| 44 | + "GRAFANA_WORKLOADS_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/main/artifacts/grafana-dashboards/eks/infrastructure/workloads.json", |
| 45 | + "GRAFANA_ISTIO_CP_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-control-plane-dashboard.json", |
| 46 | + "GRAFANA_ISTIO_MESH_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-mesh-dashboard.json", |
| 47 | + "GRAFANA_ISTIO_PERF_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-performance-dashboard.json", |
| 48 | + "GRAFANA_ISTIO_SERVICE_DASH_URL" : "https://raw.githubusercontent.com/aws-observability/aws-observability-accelerator/v0.2.0/artifacts/grafana-dashboards/eks/istio/istio-service-dashboard.json" |
| 49 | + }, |
| 50 | + "kustomizations": [ |
| 51 | + { |
| 52 | + "kustomizationPath": "./artifacts/grafana-operator-manifests/eks/infrastructure" |
| 53 | + }, |
| 54 | + { |
| 55 | + "kustomizationPath": "./artifacts/grafana-operator-manifests/eks/istio" |
| 56 | + } |
| 57 | + ] |
| 58 | + }, |
| 59 | + "Istio.pattern.enabled": true |
| 60 | + } |
| 61 | +``` |
| 62 | + |
| 63 | +Once completed the rest of the _Deploying_ steps, you can move on with the deployment of the Istio workload. |
| 64 | + |
| 65 | +## Visualization |
| 66 | + |
| 67 | +### 1. Grafana dashboards |
| 68 | + |
| 69 | +Go to the Dashboards panel of your Grafana workspace. You will see a list of Istio dashboards under the `Observability Accelerator Dashboards` |
| 70 | + |
| 71 | + |
| 72 | + |
| 73 | + |
| 74 | +Open one of the Istio dasbhoards and you will be able to view its visualization |
| 75 | + |
| 76 | + |
| 77 | + |
| 78 | + |
| 79 | +### 2. Amazon Managed Service for Prometheus rules and alerts |
| 80 | + |
| 81 | +Open the Amazon Managed Service for Prometheus console and view the details of your workspace. Under the `Rules management` tab, you will find new rules deployed. |
| 82 | + |
| 83 | + |
| 84 | + |
| 85 | + |
| 86 | +!!! note |
| 87 | + To setup your alert receiver, with Amazon SNS, follow [this documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver.html) |
| 88 | + |
| 89 | +## Deploy an example application to visualize metrics |
| 90 | + |
| 91 | +In this section we will deploy Istio's Bookinfo sample application and extract metrics using the AWS OpenTelemetry collector. When downloading and configuring `istioctl`, there are samples included in the Istio package directory. The deployment files for Bookinfo are found in the `samples` folder. Additional details can be found on Istio's [Getting Started](https://istio.io/latest/docs/setup/getting-started/) documentation |
| 92 | + |
| 93 | +### 1. Deploy the Bookinfo Application |
| 94 | + |
| 95 | +1. Using the AWS CLI, configure kubectl so you can connect to your EKS cluster. Update for your region and EKS cluster name |
| 96 | +```sh |
| 97 | +aws eks update-kubeconfig --region <enter-your-region> --name <cluster-name> |
| 98 | +``` |
| 99 | +2. Label the default namespace for automatic Istio sidecar injection |
| 100 | +```sh |
| 101 | +kubectl label namespace default istio-injection=enabled |
| 102 | +``` |
| 103 | +3. Navigate to the Istio folder location. For example, if using Istio v1.18.2 in Downloads folder: |
| 104 | +```sh |
| 105 | +cd ~/Downloads/istio-1.18.2 |
| 106 | +``` |
| 107 | +4. Deploy the Bookinfo sample application |
| 108 | +```sh |
| 109 | +kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml |
| 110 | +``` |
| 111 | +5. Connect the Bookinfo application with the Istio gateway |
| 112 | +```sh |
| 113 | +kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml |
| 114 | +``` |
| 115 | +6. Validate that there are no issues with the Istio configuration |
| 116 | +```sh |
| 117 | +istioctl analyze |
| 118 | +``` |
| 119 | +7. Get the DNS name of the load balancer for the Istio gateway |
| 120 | +```sh |
| 121 | +GATEWAY_URL=$(kubectl get svc istio-ingressgateway -n istio-system -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}') |
| 122 | +``` |
| 123 | + |
| 124 | +### 2. Generate traffic for the Istio Bookinfo sample application |
| 125 | + |
| 126 | +For the Bookinfo sample application, visit `http://$GATEWAY_URL/productpage` in your web browser. To see trace data, you must send requests to your service. The number of requests depends on Istio’s sampling rate and can be configured using the Telemetry API. With the default sampling rate of 1%, you need to send at least 100 requests before the first trace is visible. To send a 100 requests to the productpage service, use the following command: |
| 127 | +```sh |
| 128 | +for i in $(seq 1 100); do curl -s -o /dev/null "http://$GATEWAY_URL/productpage"; done |
| 129 | +``` |
| 130 | + |
| 131 | +### 3. Explore the Istio dashboards |
| 132 | + |
| 133 | +Log back into your Amazon Managed Grafana workspace and navigate to the dashboard side panel. Click on the `Observability Accelerator Dashboards` folder and open the `Istio Service` Dashboard. Use the Service dropdown menu to select the `reviews.default.svc.cluster.local` service. This gives details about metrics for the service, client workloads (workloads that are calling this service), and service workloads (workloads that are providing this service). |
| 134 | + |
| 135 | + |
| 136 | + |
| 137 | + |
| 138 | +Explore the Istio Control Plane, Mesh, and Performance dashboards as well. |
| 139 | + |
| 140 | +Control Plane |
| 141 | + |
| 142 | + |
| 143 | + |
| 144 | + |
| 145 | +Mesh |
| 146 | + |
| 147 | + |
| 148 | +Performance |
| 149 | + |
| 150 | + |
| 151 | + |
| 152 | + |
| 153 | +## Teardown |
| 154 | + |
| 155 | +You can teardown the whole CDK stack with the following command: |
| 156 | + |
| 157 | +```bash |
| 158 | +make pattern single-new-eks-opensource-observability destroy |
| 159 | +``` |
0 commit comments