|
| 1 | +# Enable ODM distributed tracing with Microprofile telemetry |
| 2 | + |
| 3 | +When applications are made observable, operations teams can more easily identify and understand the root causes of bugs, bottlenecks, and other inefficiencies. Liberty offers a robust framework for developing such observable applications and integrates seamlessly with numerous third-party monitoring tools. |
| 4 | + |
| 5 | +In the [Monitor ODM liberty metrics with mpMetrics and Prometheus](https://github.com/DecisionsDev/odm-docker-kubernetes/blob/opentelemetry/contrib/monitor/mpmetrics/README.md) tutorial, we detailed how to enable Liberty metrics that depict the internal state of various Liberty components. In this document, we will discuss how to utilize MicroProfile Telemetry, which assists in collecting data on the paths that application requests take through services. More details on the usage of Microprofile Telemetry can be found in the [Liberty documentation](https://openliberty.io/docs/latest/microprofile-telemetry.html). |
| 6 | + |
| 7 | +The goal of this tutorial is to demonstrate how to configure ODM on Kubernetes to enable communication with an OpenTelemetry collector that can process generated traces. This is not an in-depth OpenTelemetry tutorial. Therefore, it is advisable to familiarize yourself with the [Open Telemetry liberty configuration](https://openliberty.io/docs/latest/microprofile-telemetry.html#ol-config) before proceeding with this tutorial. |
| 8 | + |
| 9 | + |
| 10 | + |
| 11 | +## Install Jaeger to display traces |
| 12 | + |
| 13 | +Jaeger will be used to display traces emitted by the Open Telemetry Java agent and collected by the OpenTelemetry (OTEL) collector. |
| 14 | + |
| 15 | +Jaeger will be installed using the [OpenShift Jaeger Operator](https://docs.openshift.com/container-platform/4.15/observability/distr_tracing/distr_tracing_jaeger/distr-tracing-jaeger-installing.html#distr-tracing-jaeger-operator-install_dist-tracing-jaeger-installing). |
| 16 | + |
| 17 | +Jaeger can be installed on various platforms, including OpenShift through the use of the [OpenShift Jaeger Operator](https://docs.openshift.com/container-platform/4.15/observability/distr_tracing/distr_tracing_jaeger/distr-tracing-jaeger-installing.html#distr-tracing-jaeger-operator-install_dist-tracing-jaeger-installing). |
| 18 | + |
| 19 | +For installations on other platforms, refer to the [Jaeger documentation](https://www.jaegertracing.io/docs/1.56/operator/) for comprehensive guidance on deploying Jaeger using its operator. |
| 20 | + |
| 21 | +## Deploy the OpenTelemetry Collector |
| 22 | + |
| 23 | +We used the following [descriptor](https://github.com/open-telemetry/opentelemetry-go/blob/main/example/otel-collector/k8s/otel-collector.yaml) as the basis for the OTEL Collector deployment. |
| 24 | +However, it's likely that you will encounter an error similar to: |
| 25 | + |
| 26 | + ```console |
| 27 | +2023-07-06T17:28:37.520Z debug jaegerexporter@v0.80.0/exporter.go:106 failed to push trace data to Jaeger {"kind": "exporter", "data_type": "traces", "name": "jaeger", "error": "rpc error: code = Unimplemented desc = unknown service jaeger.api_v2.CollectorService"} |
| 28 | + ``` |
| 29 | + |
| 30 | +A solution is provided in the following [article](https://cloudbyt.es/blog/switching-to-jaeger-otel-collector). |
| 31 | + |
| 32 | +You can also utilize the [otel-collector.yaml](./otel-collector.yaml) file we used for this tutorial by applying it with: |
| 33 | + |
| 34 | + ```bash |
| 35 | +kubectl apply -f otel-collector.yaml |
| 36 | + ``` |
| 37 | + |
| 38 | +Verify that the OpenTelemetry Collector is up and running by executing: |
| 39 | + |
| 40 | + ```bash |
| 41 | +kubectl logs deployment/otel-collector |
| 42 | + ``` |
| 43 | + |
| 44 | +You should get the message : |
| 45 | + |
| 46 | + ```console |
| 47 | +"Everything is ready. Begin running and processing data." |
| 48 | + ``` |
| 49 | + |
| 50 | +## Install ODM with the Open Telemetry agent |
| 51 | + |
| 52 | +In this tutorial, we will inject the OpenTelemetry java agent inside the Decision Server Runtime and configure it to communicate with the OTEL Collector using JVM options. Then, we will manage some execution to generate traces and inspect them with the Jaeger UI. |
| 53 | + |
| 54 | +### Prepare your environment for the ODM installation (5 min) |
| 55 | + |
| 56 | +To access the ODM material, you need an IBM entitlement key to pull images from the IBM Cloud Container registry. |
| 57 | +This key will be utilized in the subsequent step of this tutorial. |
| 58 | + |
| 59 | +#### a. Retrieve your entitled registry key |
| 60 | + |
| 61 | +- Log in to [MyIBM Container Software Library](https://myibm.ibm.com/products-services/containerlibrary) with the IBMid and password that are associated with the entitled software. |
| 62 | + |
| 63 | +- In the **Container Software and Entitlement Keys** tile, verify your entitlement on the **View library page**, and then go to *Entitlement keys* to retrieve the key. |
| 64 | + |
| 65 | +#### b. Create a pull secret by running the kubectl create secret command |
| 66 | + |
| 67 | +```bash |
| 68 | +kubectl create secret docker-registry my-odm-docker-registry --docker-server=cp.icr.io \ |
| 69 | + --docker-username=cp --docker-password="<ENTITLEMENT_KEY>" --docker-email=<USER_EMAIL> |
| 70 | +``` |
| 71 | + |
| 72 | +Where: |
| 73 | +* `<ENTITLEMENT_KEY>` is the entitlement key from the previous step. Make sure you enclose the key in double-quotes. |
| 74 | +* `<USER_EMAIL>` is the email address associated with your IBMid. |
| 75 | + |
| 76 | +> **Note** |
| 77 | +> The `cp.icr.io` value for the docker-server parameter is the only registry domain name that contains the images. You must set the docker-username to `cp` to use an entitlement key as docker-password. |
| 78 | +
|
| 79 | +The my-odm-docker-registry secret name is already used for the `image.pullSecrets` parameter when you run a helm install of your containers. The `image.repository` parameter is also set by default to `cp.icr.io/cp/cp4a/odm`. |
| 80 | + |
| 81 | +#### c. Add the public IBM Helm charts repository |
| 82 | + |
| 83 | +```bash |
| 84 | +helm repo add ibm-helm https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm |
| 85 | +helm repo update |
| 86 | +``` |
| 87 | + |
| 88 | +#### d. Check your access to the ODM chart |
| 89 | + |
| 90 | +```bash |
| 91 | +$ helm search repo ibm-odm-prod |
| 92 | +NAME CHART VERSION APP VERSION DESCRIPTION |
| 93 | +ibm-helm/ibm-odm-prod 24.0.0 9.0.0.0 IBM Operational Decision Manager |
| 94 | +``` |
| 95 | + |
| 96 | +### Install an IBM Operational Decision Manager release (10 min) |
| 97 | + |
| 98 | +Install a Kubernetes release with the default configuration named `otel-odm-release`, injecting the OTEL Java agent with the relevant JVM configuration. |
| 99 | + |
| 100 | +We'll use the **decisionServerRuntime.downloadUrl** parameter to download the [OTEL Java agent](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.32.1/opentelemetry-javaagent.jar), which will be injected into the container at the `/config/download/opentelemetry-javaagent.jar` path. |
| 101 | + |
| 102 | +To configure the OTEL Java agent, we need to set up some JVM options, such as: |
| 103 | + |
| 104 | +```bash |
| 105 | + -javaagent:/config/download/opentelemetry-javaagent.jar |
| 106 | + -Dotel.sdk.disabled=false |
| 107 | + -Dotel.exporter.otlp.endpoint=http://otel-collector.otel.svc.cluster.local:4317 |
| 108 | + -Dotel.service.name=odm |
| 109 | + -Dotel.traces.exporter=otlp |
| 110 | + -Dotel.logs.exporter=none |
| 111 | + -Dotel.metrics.exporter=none |
| 112 | +``` |
| 113 | + |
| 114 | +To do this, create the **otel-runtime-jvm-options-configmap** configmap that will be associated to the **decisionServerRuntime.jvmOptionsRef** parameter : |
| 115 | + |
| 116 | +```bash |
| 117 | +kubectl create -f otel-runtime-jvm-options-configmap.yaml |
| 118 | +``` |
| 119 | + |
| 120 | +We will also add a parameter to add some liberty configurations that could be increase some traces using the **decisionServerRuntime.libertyHookRef** parameter. |
| 121 | +Create the following secret using the libertyHookEnd.xml file : |
| 122 | + |
| 123 | +```bash |
| 124 | +kubectl create secret generic runtime-liberty-configuration --from-file=libertyHookEnd.xml |
| 125 | +``` |
| 126 | + |
| 127 | + |
| 128 | +Then, install the ODM release : |
| 129 | + |
| 130 | +```bash |
| 131 | +helm install otel-odm-release ibm-helm/ibm-odm-prod \ |
| 132 | + --set image.repository=cp.icr.io/cp/cp4a/odm --set image.pullSecrets=icregistry-secret \ |
| 133 | + --set license=true --set usersPassword=odmAdmin \ |
| 134 | + --set internalDatabase.persistence.enabled=false --set internalDatabase.populateSampleData=true \ |
| 135 | + --set internalDatabase.runAsUser='' --set customization.runAsUser='' --set service.enableRoute=true \ |
| 136 | + --set decisionServerRuntime.downloadUrl='{https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.32.1/opentelemetry-javaagent.jar}' \ |
| 137 | + --set decisionServerRuntime.jvmOptionsRef=otel-runtime-jvm-options-configmap \ |
| 138 | + --set decisionServerRuntime.libertyHookRef=runtime-liberty-configuration |
| 139 | +``` |
| 140 | + |
| 141 | +Having a look at the Decision Server Runtime pod logs, you should see : |
| 142 | + |
| 143 | +```console |
| 144 | +[otel.javaagent 2024-04-03 18:03:27:166 +0200] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.1 |
| 145 | +``` |
| 146 | + |
| 147 | +Using **-Dotel.traces.exporter=otlp** JVM options, no OTEL traces are exported in the log files. So, that's normal to see nothing here. If you need to display them, you can replace it by **-Dotel.traces.exporter=logging** |
| 148 | + |
| 149 | +## Generate some traces and observe them using the Jaegger UI |
| 150 | + |
| 151 | +### Execute some runtime call |
| 152 | + |
| 153 | +After instantiating ODM by populating it with the sample data, we are ready to directly execute some Decision Server Runtime calls. |
| 154 | + |
| 155 | +Refer to [this documentation](https://www.ibm.com/docs/en/odm/8.12.0?topic=tasks-configuring-external-access) to retrieve the endpoints. |
| 156 | + |
| 157 | +For example, on OpenShift, you can obtain the route names and hosts with the following commands: |
| 158 | + |
| 159 | + ```bash |
| 160 | + kubectl get routes --no-headers --output custom-columns=":metadata.name,:spec.host" |
| 161 | + ``` |
| 162 | + |
| 163 | + You get the following hosts: |
| 164 | + ```console |
| 165 | + my-odm-release-odm-dc-route <DC_HOST> |
| 166 | + my-odm-release-odm-dr-route <DR_HOST> |
| 167 | + my-odm-release-odm-ds-console-route <DS_CONSOLE_HOST> |
| 168 | + my-odm-release-odm-ds-runtime-route <DS_RUNTIME_HOST> |
| 169 | + ``` |
| 170 | + |
| 171 | +You perform a basic authentication ODM runtime call in the following way: |
| 172 | + |
| 173 | + ```bash |
| 174 | + curl -H "Content-Type: application/json" -k --data @payload.json \ |
| 175 | + -H "Authorization: Basic b2RtQWRtaW46b2RtQWRtaW4=" \ |
| 176 | + https://<DS_RUNTIME_HOST>/DecisionService/rest/production_deployment/1.0/loan_validation_production/1.0 |
| 177 | + ``` |
| 178 | + |
| 179 | + Where `b2RtQWRtaW46b2RtQWRtaW4=` is the base64 encoding of the current username:password odmAdmin:odmAdmin |
| 180 | + |
| 181 | +### Observe the collected traces on the Jaegger UI |
| 182 | + |
| 183 | +If you followed the standard Jaeger installation using the OpenShift Operator, the Jaeger all-in-one instance should be accessible via a route named `<jaeger-all-in-one.XXX>`. |
| 184 | + |
| 185 | +To observe Decision Server Runtime executions in the Jaeger UI, navigate to this route, click on the "Search" menu, and retrieve information about the previous executions. |
| 186 | +You will need to select or enter **odm** as the Service name and select **POST /DecisionService/rest/* ** as the Operation. Then, click on the **Find Traces** button. |
| 187 | + |
| 188 | + |
| 189 | + |
| 190 | +By clicking on a **odm:POST /DecisionService/rest/** result, you can access detailed information about the execution: |
| 191 | + |
| 192 | + |
| 193 | + |
| 194 | +To gain more insights into span details, you can add additional Liberty features. |
| 195 | +Edit the `runtime-liberty-configuration` secret and uncomment the lines that incorporate Liberty OpenTelemetry features: |
| 196 | + |
| 197 | + |
| 198 | + ```xml |
| 199 | +<server> |
| 200 | + <featureManager> |
| 201 | + <feature>servlet-4.0</feature> |
| 202 | + <feature>appSecurity-3.0</feature> |
| 203 | + <feature>jsp-2.3</feature> |
| 204 | + <feature>jdbc-4.1</feature> |
| 205 | + <feature>ldapRegistry-3.0</feature> |
| 206 | + <feature>openidConnectClient-1.0</feature> |
| 207 | + <feature>transportSecurity-1.0</feature> |
| 208 | + <feature>monitor-1.0</feature> |
| 209 | + <feature>mpOpenAPI-2.0</feature> |
| 210 | + <feature>mpOpenTracing-2.0</feature> |
| 211 | + <feature>microProfile-4.0</feature> |
| 212 | + </featureManager> |
| 213 | +</server> |
| 214 | + ``` |
| 215 | + |
| 216 | +The Decision Server Runtime logs must show : |
| 217 | + |
| 218 | + ```console |
| 219 | +[4/3/24, 18:36:23:612 CEST] 0000003b FeatureManage A CWWKF1037I: The server added the [jaxrs-2.1, jaxrsClient-2.1, jsonb-1.0, jsonp-1.1, jwt-1.0, microProfile-4.0, mpConfig-2.0, mpFaultTolerance-3.0, mpHealth-3.0, mpJwt-1.2, mpMetrics-3.0, mpOpenAPI-2.0, mpOpenTracing-2.0, mpRestClient-2.0, opentracing-2.0] features to the existing feature set. |
| 220 | +[4/3/24, 18:36:23:613 CEST] 0000003b FeatureManage A CWWKF0012I: The server installed the following features: [appSecurity-2.0, appSecurity-3.0, cdi-2.0, distributedMap-1.0, el-3.0, federatedRegistry-1.0, jaxrs-2.1, jaxrsClient-2.1, jdbc-4.1, jndi-1.0, json-1.0, jsonb-1.0, jsonp-1.1, jsp-2.3, jwt-1.0, ldapRegistry-3.0, microProfile-4.0, monitor-1.0, mpConfig-2.0, mpFaultTolerance-3.0, mpHealth-3.0, mpJwt-1.2, mpMetrics-3.0, mpOpenAPI-2.0, mpOpenTracing-2.0, mpRestClient-2.0, oauth-2.0, openidConnectClient-1.0, opentracing-2.0, servlet-4.0, ssl-1.0, transportSecurity-1.0]. |
| 221 | +[4/3/24, 18:36:23:614 CEST] 0000003b FeatureManage A CWWKF0008I: Feature update completed in 2.964 seconds. |
| 222 | + ``` |
| 223 | + |
| 224 | +Now, when running new Runtime Execution, you should see more managed span in Jaegger UI : |
| 225 | + |
| 226 | + |
| 227 | + |
| 228 | +If you want to add more span in the java code to observe code behaviour, have a look at the [liberty tutorial](https://openliberty.io/guides/microprofile-telemetry-jaeger.html) |
| 229 | + |
| 230 | + |
0 commit comments