traceidratio vs parentbased_traceidratio

[itsmaneesh]

Hi,
Can you please clarify the sampling behavior of parentbased_traceidratio compared to traceidratio sampler strategy options: (for OTEL_TRACES_SAMPLER env variable) with OpenTelemetry auto-instrumentation.

I have gone through couple of discussions threads and understood that parentbased_traceidratio strategy will sample the traces (as per OTEL_TRACES_SAMPLER_ARG value) at upstream level (typically at the time of trace Id creation) and ensure the completeness of the trace (meaning, all the expected spans will be associated with the trace)

Discussion threads:
open-telemetry/opentelemetry-collector-contrib#4462
open-telemetry/opentelemetry-java#3235

It would be great if you can point to me to any doc or clarify here the exact difference between parentbased_traceidratio and traceidratio

Thank you,
Maneesh

[mateuszrzeszutek]

Hey @itsmaneesh

You can think of parentbased_ as a decorator over the sampler referenced after the underscore (that is, traceidratio in this case).
Take a look at the official samplers doc, in particular the table that describes the behavior of the sampler depending on the input.
If the span being sampled is the root span of the trace, parentbased_traceidratio will behave exactly the same as traceidratio; if it's not the root of the trace, then you can look up how it'll behave in the table.

[laurit]

You can view the parentbased one as sampling the whole trace, the traces you'll end up with are complete. The non parent based one will sample individual spans, so you could end up with a bunch of incomplete traces. Use the parentbased one unless you know what you are doing.

[itsmaneesh]

@laurit I had a look at the class to understand the algorithm.
But, have couple of questions:

1. Algorithm seem to extract absolute long value out of the traceid hexadecimal (base16) string from offset 16 (mid of the string length) till end and compare that long value against the (long) (ratio * Long.MAX_VALUE) where ratio defined with env variable OTEL_TRACES_SAMPLER_ARG and 0.0 < ratio < 1.0. Question is how does this ensure 10% of traces will be sampled, provided the 0.1 is the ratio.?
2. From your statement:

with OTEL_TRACES_SAMPLER_ARG as 0.1 you could say that given large enough time window it is likely that 10% of traces will be sampled

what is the "given large enough time window" ? Can you please clarify.

[laurit]

Trace id is 128 bits/16 bytes. My understanding of this is that given a uniformly distributed trace id its upper and lower half also have uniform distribution. So we can use only half of the trace id which is more convenient to deal with. To simplify the calculations we don't want to deal with negative numbers so we use Math.abs after which we are left with positive numbers and Long.MIN_VALUE as the only negative number. As there is no positive value that corresponds to Long.MIN_VALUE it is its own absolute value, we'll deal with this anomaly by ignoring it and making it so that for our purposes Long.MIN_VALUE is the same as zero. Now we have reduced trace id to a number that is in rang of 0 to Long.MAX_VALUE and the assumption is that it is still uniformly distributed. So if we want to get 10% of the traces we can compute 0.1*Long.MAX_VALUE and pick traces where the reduced trace id is below that value.

what is the "given large enough time window" ? Can you please clarify.

You could think of this as an analogy of tossing a coin. Even if you start off with 5 straight heads eventually, after you have repeated the toss enough times, you'll have roughly the same amount of heads and tails.

[itsmaneesh]

@laurit Will test this configuration with OTEL_TRACES_SAMPLER_ARG as 0.1 and let you know if there is any deviation from our expectation of 10 % traces to be sampled. Thank you.

But, If we have two services ServiceA (upstream) and ServiceB (downstream) (both are using auto-instrumentation) have different OTEL_TRACES_SAMPLER_ARG values then how does the sampling strategy works:

For e.g.:
Scenaio#1: Upstream system has lower value for OTEL_TRACES_SAMPLER_ARG
ServiceA (upstream) has OTEL_TRACES_SAMPLER_ARG as 0.1 and ServiceB (downstream) has OTEL_TRACES_SAMPLER_ARG as 0.3.

Does the lower value of upstream system takes precedence and entire sampling will be based on value 0.1 i.e. 10 % traces sampled.?
Please confirm.

Scenario#2: Downstream system has lower value for OTEL_TRACES_SAMPLER_ARG
ServiceA(upstream) has OTEL_TRACES_SAMPLER_ARG as 0.3 and ServiceB(downstream) has OTEL_TRACES_SAMPLER_ARG as 0.1.

Does the upstream system value takes precedence despite being a higher value than the downstream system value? i.e.: 30 % traces will be sampled through out flow from ServiceA -> ServiceB. (downstream system's lower sampling value will be ignored as the upstream system's sampling value takes precedence?
Please clarify.

[laurit]

If you are using parent based sampler then the sampling decision will be done by the service that creates the trace (ServiceA).
If you are not using parent based sampler then for scenario#1 the lower value from ServiceA should take precedence. This is because the algorithm used in java traceidratio sampler is deterministic. All traces that are sampled with 0.1 are also sampled with 0.3. I have no idea what would happen if you mix services written with java with services written in some other language. For scenario#2 the lower value of ServiceB will take effect and some traces will be missing spans from ServiceB.
At least this is how I understand it.

[trask]

@itsmaneesh also check out https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#requirements-for-traceidratiobased-sampler-algorithm