GC overhead limit exceeded

[JHub-Wei]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

After the OAP is started for a while, the instance is abnormal and the error java.lang.OutOfMemoryError: GC overhead limit exceeded is reported. The VM memory is 8 GB. The JVM memory of the Docker container is set to 5 GB, and it could not be fixed.

What you expected to happen

The OAP can be normally started and written to the ES.

How to reproduce

Version 8.4.0 used

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

If you want to discuss about OOM, at least you should provide the result memory analysis. Otherwise, what do you expect from the community?

[wu-sheng]

OAP server has performance improvement in every release of 8.x, so what version do you use, how you use, what your memory dump looks like are the basic context you need to provide

[JHub-Wei]

As I mentioned in my discussion, version 8.4.0 is used.Consume Kafka and write it to ES7.

[JHub-Wei]

Two pieces of memory data are dumped every two minutes after startup. The comparison shows that the number of ServiceInstanceRelationClientCpmMetrics instances increases by 64319.

[wu-sheng]

64319 seems pretty normal if you have a complex topology in instance level. Such as A service has 50 pods, and B has another 50 pods, then you simply get 2500 call relationship for just one call between 2 services.

[JHub-Wei]

Do you have any suggestions or solutions for this situation? Thank you.

[wu-sheng]

First of all, I would like to ask you to upgrade to 8.8.1. The change logs between 8.4.0 and 8.8.1, could show you that we improve a lot about data persistence. You could check whether this could help you. In another hand, if there is no leak(if you have verified GC and low load help), increasing memory to avoid full GC and OOM is important. We have seen users set memory to 20G to keep the performance high enough.

[JHub-Wei]

Okay, thank you very much.

[JHub-Wei]

[JHub-Wei]

[JHub-Wei]

[JHub-Wei]

The instance have been on the gc, after running for two or three minutes.