JavaAgent Benchmark testing

[sparam]

Hi,

I am trying to figure out a way to do benchmark testing the Otel Java agent and load test the OTel Collector.

There seems to be some good support for benchmark testing. Could these be used for Otel Java agent benchmark testing? Could these be executed independent of building the entire project?
https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/master/benchmark

Appreciate if you could point to some recommended benchmark testing.

Thanks!

[johnbley]

The ones in that directory look more like microbenchmarks of specific pieces of functionality (e.g., class loading overhead). If you're interested in whole-app studies, you might do better starting from some of the apps in smoke-tests with any load generator of your choice (e.g., ab), or any similar trivial application that contains technology relevant to your needs (e.g., jsp uses database pool and runs select pg_sleep(1) or similar).

[trask]

also check out https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/benchmark-e2e in case that helps

[Sagar1905]

Page doesnt exist. 404

[trask]

that module has been replaced by https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/benchmark-overhead

[ecourreges-orange]

I did rough tests using jmeter and https://github.com/williewheeler/otel-demo which I felt was an easy REST+DB microservice demo which corresponds to my real world scenarios. Forked here

Test protocol

My goal was to get a rough estimate of the performance penalty of activating traces.
At 1% and 100% sampling.
No instrumentation vs opentelemetry-java-instrumentation vs spring-cloud-sleuth

• Ubuntu 18.04 VM on a Core I7 quad core (8 thread) laptop, 8G RAM
• otel-demo running with docker-compose, UI disabled, jaeger-all-in-one disabled.
• otel-java-instr version 0.14.1 (almost 1.0 version)
• spring-cloud-sleuth version 2.2.3
• When traces are activated, they are sent to a remote server
• Jmeter running locally with 16 threads to call otel-demo API on localhost:8081
• 60s, run 5 times, averaged last 3 runs to let the jvm setup its caches

Results

	Reqs/min 16threads	RTT avg for 16 jmeter threads	Penalty pct 16 threads	Reqs/min 1 thread	RTT Avg 1 thread	Penalty pct 1 thread	Penatly RTT 1 thread
No instrumentation	19400	49ms	0%	11475	5.100	0%	0ms
otel-java-instr (jaeger) 1%	16650	57ms	17%	9453	6.200	21%	1.1ms
otel-java-instr (jaeger) 100%	14550	65ms	33%	7825	7.510	46%	2.4ms
spring-cloud-sleuth (zipkin) 1%	18400	51ms	5%	10782	5.43	6%	0.3ms
spring-cloud-sleuth (zipkin) 100%	16200	59ms	20%	9269	6.33	24%	1.2ms

Missing results:
Startup time was noticeably slower with opentelemetry-java-instrumentation, somewhere from 20% to 100% slower, but no tangible results achieved.
One aspect to consider would also be how longer it takes for the jvm and agent to be "hot in cache", especially for shorter lived apps like serverless?

Personal Conclusion

otel-java-instr is usable everywhere except full time in production where you must decide if you are willing to accept up to 20% additional CPU and a minimum latecy hit of 0.5ms per traversed application.
If full time production use is required with most limited impact possible, I would probably go for spring-cloud-sleuth for spring apps, and opentelemetry-java with some work.

Addendum

I haven't decided where else to post my results, medium, or pull request to williewheeler/otel-demo, or nowhere.

[Johnny850807]

Thanks for your work !

[magicliang]

Good job and here is my question:

Performace penalty means performance decrease？

otel-java-instr (jaeger) 100%'s Penalty means with 100% otel-java-instrument, the performance will drop down by 33%? That's a huge overhead considering javaagent's efficiency.

[trask]

hi @magicliang, there's some benchmark results that we publish at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/gh-pages/benchmark-overhead/results based on https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/benchmark-overhead

but in general, we recommend just dropping the javaagent into your existing application benchmarks to get real numbers for your specific application as it can vary

[magicliang]

hi @magicliang, there's some benchmark results that we publish at https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/gh-pages/benchmark-overhead/results based on https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/benchmark-overhead

but in general, we recommend just dropping the javaagent into your existing application benchmarks to get real numbers for your specific application as it can vary

Its a detail result which has about 10-20 columns, I would study it carefully. Thanks for that.

[qianshengzj]

Thanks for your work !

[anuraaga]

Thanks for looking into performance @ecourreges-orange. We definitely need to understand the performance of this instrumentation.

Just curious, can you confirm the difference in the trace? I would expect otel to possibly have more spans since we instrument more directly with byte code manipulation. Also any trends on the number of attributes? Otel has a philosophy of sending as much data as possible, and this indeed has performance implications - we don't have good ways to control this yet unfortunately.

[ecourreges-orange]

Indeed you are right to ask, it is not a 1-1 comparison as otel returns more spans and more attributes, so it would have to be pushed a bit further for honest comparison.
As far as functionality, otel is definitely more plug and play.
Here you can see that the spring data mysql part was missed by spring cloud sleuth, event though it is supported.

otel:

sleuth: