Skip to content

Commit 01674d3

Browse files
authored
Merge pull request #77583 from gwynnemonahan/OSSM-6302
OSSM-6302 [DOC] Tempo + OTEL configuration
2 parents 61044b6 + 571430b commit 01674d3

File tree

3 files changed

+150
-49
lines changed

3 files changed

+150
-49
lines changed

modules/ossm-configuring-distr-tracing-tempo.adoc

Lines changed: 148 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -5,21 +5,76 @@ This module is included in the following assemblies:
55

66
:_mod-docs-content-type: PROCEDURE
77
[id="ossm-configuring-distr-tracing-tempo_{context}"]
8-
= Configuring the distributed tracing platform (Tempo)
8+
= Configuring the {TempoName} and the {OTELName}
99

10-
You can expose tracing data to the {TempoName} stack by appending a named element and the `zipkin` provider to the `spec.meshConfig.extensionProviders` specification in the `ServiceMehControlPlane`, as shown in the following example. Then, a telemetry custom resource configures Istio proxies to collect trace spans and send them to the Tempo distributor service endpoint.
10+
You can expose tracing data to the {TempoName} by appending a named element and the `opentelemetry` provider to the `spec.meshConfig.extensionProviders` specification in the `ServiceMeshControlPlane`. Then, a telemetry custom resource configures Istio proxies to collect trace spans and send them to the OpenTelemetry Collector endpoint.
1111

12-
You can create a TempoStack instance in the `tracing-system` namespace _after_ creating the `ServiceMeshControlPlane` and the `ServiceMeshMemberRoll` resources.
12+
//As of July 2, 2024, there is no common attribute for OpenTelemetry Collector.
13+
14+
You can create a {OTELName} instance in a mesh namespace and configure it to send tracing data to a tracing platform backend service.
1315

1416
.Prerequisites
1517

16-
* You have installed the {TempoOperator} and {SMProductName} Operator in the `openshift-operators` namespace.
17-
* You have created namespaces such as `istio-system` and `tracing-system`.
18+
* You created a TempoStack instance using the Red Hat {TempoOperator} in the `tracing-system` namespace. For more information, see "Installing {TempoName}" in the "Additional resources" section.
19+
20+
* You installed the {OTELOperator} in either the recommended namespace or the `openshift-operators` namespace. For more information, see "Installing the {OTELName}" in the "Additional resources" section.
21+
22+
* If using {SMProductName} 2.5 or earlier, set the `spec.tracing.type` parameter of the `ServiceMeshControlPlane` resource to `None` so tracing data can be sent to the OpenTelemetry Collector.
1823
1924
.Procedure
2025

21-
. Configure the `ServiceMeshControlPlane` resource to define an extension provider:
26+
. Create an OpenTelemetry Collector instance in a mesh namespace. This example uses the `bookinfo` namespace:
27+
+
28+
.Example OpenTelemetry Collector configuration
29+
[source, yaml]
30+
----
31+
apiVersion: opentelemetry.io/v1alpha1
32+
kind: OpenTelemetryCollector
33+
metadata:
34+
name: otel
35+
namespace: bookinfo # <1>
36+
annotations:
37+
sidecar.istio.io/inject: 'true' # <2>
38+
spec:
39+
mode: deployment
40+
config: |
41+
receivers:
42+
otlp:
43+
protocols:
44+
grpc:
45+
endpoint: 0.0.0.0:4317
46+
exporters:
47+
otlp:
48+
endpoint: "tempo-sample-distributor.tracing-system.svc.cluster.local:4317" # <3>
49+
tls:
50+
insecure: true
51+
service:
52+
pipelines:
53+
traces:
54+
receivers: [otlp]
55+
processors: []
56+
exporters: [otlp]
57+
----
58+
<1> Include the namespace in the `ServiceMeshMemberRoll` member list.
59+
<2> The sidecar injection annotation is only required when you enable the `spec.security.dataPlane` parameter for mTLS encryption in the `ServiceMeshControlPlane` resource.
60+
<3> In this example, a TempoStack instance is running in the `tracing-system` namespace. You do not have to include the TempoStack namespace, such as`tracing-system`, in the `ServiceMeshMemberRoll` member list.
61+
+
62+
[NOTE]
63+
====
64+
You only need to create one instance of the OpenTelemetry Collector in one of the `ServiceMeshMemberRoll` member namespaces.
65+
====
66+
67+
. Check the `otel-collector` pod log and verify that the pod is running.
68+
+
69+
.Example `otel-collector` pod log check
70+
[source,terminal]
71+
----
72+
$ oc logs -n bookinfo -l app.kubernetes.io/name=otel-collector
73+
----
74+
+
75+
. Create or update an existing `ServiceMeshControlPlane` custom resource (CR) in the `istio-system` namespace:
2276
+
77+
.Example SMCP custom resource
2378
[source,yaml]
2479
----
2580
kind: ServiceMeshControlPlane
@@ -28,36 +83,61 @@ metadata:
2883
name: basic
2984
namespace: istio-system
3085
spec:
31-
# ...
86+
addons:
87+
grafana:
88+
enabled: false
89+
kiali:
90+
enabled: true
91+
prometheus:
92+
enabled: true
3293
meshConfig:
3394
extensionProviders:
34-
- name: tempo
35-
zipkin:
36-
service: tempo-sample-distributor.tracing-system.svc.cluster.local
37-
port: 9411
38-
tracing:
39-
sampling: 10000
40-
type: None <1>
41-
version: v2.5
42-
---
43-
kind: ServiceMeshMemberRoll
44-
apiVersion: maistra.io/v1
45-
metadata:
46-
name: default
47-
namespace: istio-system
48-
spec:
49-
members:
50-
- tracing-system
95+
- name: otel
96+
opentelemetry:
97+
port: 4317
98+
service: otel-collector.bookinfo.svc.cluster.local
99+
policy:
100+
type: Istiod
101+
telemetry:
102+
type: Istiod
103+
version: v2.6
51104
----
52-
<1> The `spec.tracing.type` setting defines a deprecated distributed tracing Jaeger instance. Set `spec.tracing.type` to `None` when connecting to a TempoStack using an `extensionProvider` setting.
53105
+
54106
[NOTE]
55107
====
56-
Create a TempoStack instance _after_ creating the `ServiceMeshControlPlane` and the `ServiceMeshMemberRoll` resources.
108+
When upgrading from SMCP 2.5 to 2.6, set the `spec.tracing.type` parameter to `None`:
109+
110+
.Example SMCP `spec.tracing.type` parameter
111+
[source,yaml]
112+
----
113+
spec:
114+
tracing:
115+
type: None
116+
----
57117
====
58118

119+
. Create a Telemetry resource in the `istio-system` namespace:
120+
+
121+
.Example Telemetry resource
122+
[source,yaml]
123+
----
124+
apiVersion: telemetry.istio.io/v1alpha1
125+
kind: Telemetry
126+
metadata:
127+
name: mesh-default
128+
namespace: istio-system
129+
spec:
130+
tracing:
131+
- providers:
132+
- name: otel
133+
randomSamplingPercentage: 100
134+
----
135+
136+
. Verify the `istiod` log.
137+
59138
. Configure the Kiali resource specification to enable a Kiali workload traces dashboard. You can use the dashboard to view tracing query results.
60139
+
140+
.Example Kiali resource
61141
[source,yaml]
62142
----
63143
apiVersion: kiali.io/v1alpha1
@@ -66,35 +146,62 @@ kind: Kiali
66146
spec:
67147
external_services:
68148
tracing:
69-
query_timeout: 30
149+
query_timeout: 30 #<1>
70150
enabled: true
71151
in_cluster_url: 'http://tempo-sample-query-frontend.tracing-system.svc.cluster.local:16685'
72152
url: '[Tempo query frontend Route url]'
73-
use_grpc: true # <1>
153+
use_grpc: true # <2>
74154
----
75-
<1> If you are not using the default HTTP or gRPC port for Jaeger or Tempo, replace the `in_cluster_url:` port with your custom port.
155+
<1> The default `query_timeout` integer value is 30 seconds. If you set the value to greater than 30 seconds, you must update `.spec.server.write_timeout` in the Kiali CR and add the annotation `haproxy.router.openshift.io/timeout=50s` to the Kiali route. Both `.spec.server.write_timeout` and `haproxy.router.openshift.io/timeout=` must be greater than `query_timeout`.
156+
<2> If you are not using the default HTTP or gRPC port, replace the `in_cluster_url:` port with your custom port.
76157
+
77158
[NOTE]
78159
====
79160
Kiali 1.73 uses the Jaeger Query API, which causes a longer response time depending on Tempo resource limits. If you see a `Could not fetch spans` error message in the Kiali UI, then check your Tempo configuration or reduce the limit per query in Kiali.
80161
====
81162

82-
. Create a TempoStack instance using the Red Hat {TempoOperator} in the `tracing-system` namespace. For more information, see "Installing the distributed tracing platform (Tempo)" in the "Additional resources" section.
163+
. Send requests to your application.
83164

84-
. Apply a Telemetry custom resource for {SMProductShortName} to start the Tempo provider setting.
85-
+
165+
. Verify the `istiod` pod logs and the `otel-collector` pod logs.
166+
167+
[id="configuring-distr-tracing-tempo-mtls-encrypted-namespace_{context}"]
168+
== Configuring the {TempoName} in a mTLS encrypted Service Mesh member namespace
169+
170+
[NOTE]
171+
====
172+
You don't need this additional `DestinationRule` configuration if you created a TempoStack instance in a namespace that is not a Service Mesh member namespace.
173+
====
174+
175+
All traffic is TLS encrypted when you enable Service Mesh `dataPlane` mTLS encryption and you create a TempoStack instance in a Service Mesh member namespace such as `tracing-system-mtls`. This encryption is not expected from the Tempo distributed service and returns a TLS error.
176+
177+
To fix the TLS error, disable the TLS `trafficPolicy` by applying a `DestinationRule` for Tempo and Kiali:
178+
179+
.Example `DestinationRule` Tempo
86180
[source,yaml]
87181
----
88-
apiVersion: telemetry.istio.io/v1alpha1
89-
kind: Telemetry
182+
apiVersion: networking.istio.io/v1alpha3
183+
kind: DestinationRule
90184
metadata:
91-
name: mesh-default
92-
namespace: istio-system
185+
name: tempo
186+
namespace: tracing-system-mtls
93187
spec:
94-
tracing:
95-
- providers:
96-
- name: tempo
97-
randomSamplingPercentage: 100
188+
host: "*.tracing-system-mtls.svc.cluster.local"
189+
trafficPolicy:
190+
tls:
191+
mode: DISABLE
98192
----
99193

100-
You can also create an Istio gateway and virtual service resources to expose an {product-title} route for accessing the Tempo Jaeger Query console.
194+
.Example `DestinationRule` Kiali
195+
[source,yaml]
196+
----
197+
apiVersion: networking.istio.io/v1alpha3
198+
kind: DestinationRule
199+
metadata:
200+
name: kiali
201+
namespace: istio-system
202+
spec:
203+
host: kiali.istio-system.svc.cluster.local
204+
trafficPolicy:
205+
tls:
206+
mode: DISABLE
207+
----

modules/ossm-enabling-jaeger.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,9 @@ spec:
2222
type: Jaeger
2323
----
2424

25-
Currently, the only tracing type that is supported is `Jaeger`.
25+
In {SMProductName} 2.6, the tracing type `Jaeger` is deprecated and disabled by default.
2626

27-
Jaeger is enabled by default. To disable tracing, set `type` to `None`.
27+
In {SMProductName} 2.5 and earlier, the tracing type `Jaeger` is enabled by default. To disable `Jaeger` tracing, set the `spec.tracing.type` parameter of the `ServiceMeshControlPlane` resource to `None`.
2828

2929
The sampling rate determines how often the Envoy proxy generates a trace. You can use the sampling rate option to control what percentage of requests get reported to your tracing system. You can configure this setting based upon your traffic in the mesh and the amount of tracing data you want to collect. You configure `sampling` as a scaled integer representing 0.01% increments. For example, setting the value to `10` samples 0.1% of traces, setting the value to `500` samples 5% of traces, and a setting of `10000` samples 100% of traces.
3030

service_mesh/v2x/ossm-observability.adoc

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,6 @@ include::modules/ossm-distr-tracing.adoc[leveloffset=+1]
2222

2323
include::modules/ossm-configuring-distr-tracing-tempo.adoc[leveloffset=+2]
2424

25-
[role="_additional-resources"]
26-
.Additional resources
27-
ifndef::openshift-rosa,openshift-dedicated[]
28-
xref:../../observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-installing.adoc[Installing the distributed tracing platform (Tempo)].
29-
endif::openshift-rosa,openshift-dedicated[]
30-
3125
include::modules/ossm-config-external-jaeger.adoc[leveloffset=+2]
3226

3327
include::modules/ossm-config-sampling.adoc[leveloffset=+2]

0 commit comments

Comments
 (0)