Skip to content

Commit 41e2ee2

Browse files
kcarmichael08kcarmichael08
authored
Merge pull request #88970 from max-cx/OBSDOCS-1356
OBSDOCS-1356: Document how to verify if Instrumentation was injected correctly
2 parents 5e06a58 + afeb3d1 commit 41e2ee2

7 files changed

+162
-7
lines changed

‎modules/otel-autoinstrumentation.adoc

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-configuration-of-instrumentation.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="otel-autoinstrumentation_{context}"]
7+
= Auto-instrumentation in the {OTELOperator}
8+
9+
Auto-instrumentation in the {OTELOperator} can automatically instrument an application without manual code changes. Developers and administrators can monitor applications with minimal effort and changes to the existing codebase.
10+
11+
Auto-instrumentation runs as follows:
12+
13+
. The {OTELOperator} injects an init-container, or a sidecar container for Go, to add the instrumentation libraries for the programming language of the instrumented application.
14+
15+
. The {OTELOperator} sets the required environment variables in the application's runtime environment. These variables configure the auto-instrumentation libraries to collect traces, metrics, and logs and send them to the appropriate OpenTelemetry Collector or another telemetry backend.
16+
17+
. The injected libraries automatically instrument your application by connecting to known frameworks and libraries, such as web servers or database clients, to collect telemetry data. The source code of the instrumented application is not modified.
18+
19+
. Once the application is running with the injected instrumentation, the application automatically generates telemetry data, which is sent to a designated OpenTelemetry Collector or an external OTLP endpoint for further processing.
20+
21+
Auto-instrumentation enables you to start collecting telemetry data quickly without having to manually integrate the OpenTelemetry SDK into your application code. However, some applications might require specific configurations or custom manual instrumentation.

‎modules/otel-config-instrumentation.adoc

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,9 @@
88

99
The {OTELName} can inject and configure the OpenTelemetry auto-instrumentation libraries into your workloads. Currently, the project supports injection of the instrumentation libraries from Go, Java, Node.js, Python, .NET, and the Apache HTTP Server (`httpd`).
1010

11-
Auto-instrumentation in OpenTelemetry refers to the capability where the framework automatically instruments an application without manual code changes. This enables developers and administrators to get observability into their applications with minimal effort and changes to the existing codebase.
12-
1311
[IMPORTANT]
1412
====
15-
The {OTELName} Operator only supports the injection mechanism of the instrumentation libraries but does not support instrumentation libraries or upstream images. Customers can build their own instrumentation images or use community images.
13+
The {OTELOperator} only supports the injection mechanism of the instrumentation libraries but does not support instrumentation libraries or upstream images. Customers can build their own instrumentation images or use community images.
1614
====
1715

1816
[id="otel-instrumentation-options_{context}"]

‎modules/otel-troubleshoot-logging-exporter-stdout.adoc renamed to ‎modules/otel-troubleshoot-debug-exporter-stdout.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@
44

55
:_mod-docs-content-type: PROCEDURE
66
[id="debug-exporter-to-stdout_{context}"]
7-
= Debug exporter
7+
= Debug Exporter
88

9-
You can configure the debug exporter to export the collected data to the standard output.
9+
You can configure the Debug Exporter to export the collected data to the standard output.
1010

1111
.Procedure
1212

‎modules/otel-troubleshooting-instrumentation-injection-into-your-workload.adoc

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-troubleshooting.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="otel-troubleshooting-instrumentation-injection-into-your-workload_{context}"]
7+
= Troubleshooting instrumentation injection into your workload
8+
9+
To troubleshoot instrumentation injection, you can perform the following activities:
10+
11+
* Checking if the `Instrumentation` object was created
12+
* Checking if the init-container started
13+
* Checking if the resources were deployed in the correct order
14+
* Searching for errors in the Operator logs
15+
* Double-checking the pod annotations
16+
17+
.Procedure
18+
19+
. Run the following command to verify that the `Instrumentation` object was successfully created:
20+
+
21+
[source,terminal]
22+
----
23+
$ oc get instrumentation -n <workload_project> # <1>
24+
----
25+
<1> The namespace where the instrumentation was created.
26+
27+
. Run the following command to verify that the `opentelemetry-auto-instrumentation` init-container successfully started, which is a prerequisite for instrumentation injection into workloads:
28+
+
29+
[source,terminal]
30+
----
31+
$ oc get events -n <workload_project> # <1>
32+
----
33+
<1> The namespace where the instrumentation is injected for workloads.
34+
+
35+
.Example output
36+
[source,terminal]
37+
----
38+
... Created container opentelemetry-auto-instrumentation
39+
... Started container opentelemetry-auto-instrumentation
40+
----
41+
42+
. Verify that the resources were deployed in the correct order for the auto-instrumentation to work correctly. The correct order is to deploy the `Instrumentation` custom resource (CR) before the application. For information about the `Instrumentation` CR, see the section "Configuring the instrumentation".
43+
+
44+
[NOTE]
45+
====
46+
When the pod starts, the {OTELOperator} checks the `Instrumentation` CR for annotations containing instructions for injecting auto-instrumentation. Generally, the Operator then adds an init-container to the application’s pod that injects the auto-instrumentation and environment variables into the application's container. If the `Instrumentation` CR is not available to the Operator when the application is deployed, the Operator is unable to inject the auto-instrumentation.
47+
====
48+
+
49+
Fixing the order of deployment requires the following steps:
50+
51+
.. Update the instrumentation settings.
52+
.. Delete the instrumentation object.
53+
.. Redeploy the application.
54+
55+
. Run the following command to inspect the Operator logs for instrumentation errors:
56+
+
57+
[source,terminal]
58+
----
59+
$ oc logs -l app.kubernetes.io/name=opentelemetry-operator --container manager -n openshift-opentelemetry-operator --follow
60+
----
61+
62+
. Troubleshoot pod annotations for the instrumentations for a specific programming language. See the required annotation fields and values in "Configuring the instrumentation".
63+
64+
.. Verify that the application pods that you are instrumenting are labeled with correct annotations and the appropriate auto-instrumentation settings have been applied.
65+
+
66+
.Example
67+
----
68+
instrumentation.opentelemetry.io/inject-python="true"
69+
----
70+
+
71+
.Example command to get pod annotations for an instrumented Python application
72+
[source,terminal]
73+
----
74+
$ oc get pods -n <workload_project> -o jsonpath='{range .items[?(@.metadata.annotations["instrumentation.opentelemetry.io/inject-python"]=="true")]}{.metadata.name}{"\n"}{end}'
75+
----
76+
77+
.. Verify that the annotation applied to the instrumentation object is correct for the programming language that you are instrumenting.
78+
79+
.. If there are multiple instrumentations in the same namespace, specify the name of the `Instrumentation` object in their annotations.
80+
+
81+
.Example
82+
----
83+
instrumentation.opentelemetry.io/inject-nodejs: "<instrumentation_object>"
84+
----
85+
86+
.. If the `Instrumentation` object is in a different namespace, specify the namespace in the annotation.
87+
+
88+
.Example
89+
----
90+
instrumentation.opentelemetry.io/inject-nodejs: "<other_namespace>/<instrumentation_object>"
91+
----
92+
93+
.. Verify that the `OpenTelemetryCollector` custom resource specifies the auto-instrumentation annotations under `spec.template.metadata.annotations`. If the auto-instrumentation annotations are in `spec.metadata.annotations` instead, move them into `spec.template.metadata.annotations`.

‎modules/otel-troubleshooting-telemetry-data-generation-by-the-instrumentation-libraries.adoc

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-troubleshooting.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="otel-troubleshooting-telemetry-data-generation-by-the-instrumentation-libraries_{context}"]
7+
= Troubleshooting telemetry data generation by the instrumentation libraries
8+
9+
You can troubleshoot telemetry data generation by the instrumentation libraries by checking the endpoint, looking for errors in your application logs, and verifying that the Collector is receiving the telemetry data.
10+
11+
.Procedure
12+
13+
. Verify that the instrumentation is transmitting data to the correct endpoint:
14+
+
15+
[source,terminal]
16+
----
17+
$ oc get instrumentation <instrumentation_name> -n <workload_project> -o jsonpath='{.spec.endpoint}'
18+
----
19+
+
20+
The default endpoint `+http://localhost:4317+` for the `Instrumentation` object is only applicable to a Collector instance that is deployed as a sidecar in your application pod. If you are using an incorrect endpoint, correct it by editing the `Instrumentation` object and redeploying your application.
21+
22+
. Inspect your application logs for error messages that might indicate that the instrumentation is malfunctioning:
23+
+
24+
[source,terminal]
25+
----
26+
$ oc logs <application_pod> -n <workload_project>
27+
----
28+
29+
. If the application logs contain error messages that indicate that the instrumentation might be malfunctioning, install the OpenTelemetry SDK and libraries locally. Then run your application locally and troubleshoot for issues between the instrumentation libraries and your application without {product-title}.
30+
31+
. Use the Debug Exporter to verify that the telemetry data is reaching the destination OpenTelemetry Collector instance. For more information, see "Debug Exporter".

‎observability/otel/otel-configuration-of-instrumentation.adoc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ include::_attributes/common-attributes.adoc[]
77
toc::[]
88

99

10-
The {OTELName} Operator uses a custom resource definition (CRD) file that defines the configuration of the instrumentation.
10+
The {OTELName} Operator uses an `Instrumentation` custom resource that defines the configuration of the instrumentation.
1111

12+
include::modules/otel-autoinstrumentation.adoc[leveloffset=+1]
1213
include::modules/otel-config-instrumentation.adoc[leveloffset=+1]

‎observability/otel/otel-troubleshooting.adoc

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ The OpenTelemetry Collector offers multiple ways to measure its health as well a
1111
include::modules/otel-troubleshoot-collecting-diagnostic-data-from-command-line.adoc[leveloffset=+1]
1212
include::modules/otel-troubleshoot-collector-logs.adoc[leveloffset=+1]
1313
include::modules/otel-troubleshoot-metrics.adoc[leveloffset=+1]
14-
include::modules/otel-troubleshoot-logging-exporter-stdout.adoc[leveloffset=+1]
14+
include::modules/otel-troubleshoot-debug-exporter-stdout.adoc[leveloffset=+1]
1515
include::modules/otel-troubleshoot-network-traffic.adoc[leveloffset=+1]
1616

1717
[role="_additional-resources"]
@@ -20,3 +20,14 @@ include::modules/otel-troubleshoot-network-traffic.adoc[leveloffset=+1]
2020
* xref:../../observability/network_observability/installing-operators.adoc#installing-network-observability-operators[Installing the Network Observability Operator]
2121
2222
* xref:../../observability/network_observability/observing-network-traffic.adoc#nw-observe-network-traffic[Observing the network traffic from the Topology view]
23+
24+
[id="troubleshooting_instrumentation_{context}"]
25+
== Troubleshooting the instrumentation
26+
27+
To troubleshoot the instrumentation, look for any of the following issues:
28+
29+
* Issues with instrumentation injection into your workload
30+
* Issues with data generation by the instrumentation libraries
31+
32+
include::modules/otel-troubleshooting-instrumentation-injection-into-your-workload.adoc[leveloffset=+2]
33+
include::modules/otel-troubleshooting-telemetry-data-generation-by-the-instrumentation-libraries.adoc[leveloffset=+2]

0 commit comments

Comments
 (0)