IoTDB is not collecting data - java.lang.OutOfMemoryError: Java heap space

[rsupreet]

OS: Linux CentOS 7
RAM: 16GB
Software version installed: Apache IoTDB - [All-in-one] - 1.0.1

I am encountering the following error(java.lang.OutOfMemoryError: Java heap space) while running the instance of IoTDB.
How can I resolve this?

2023-06-14 12:45:19,899 [pool-47-IoTDB-instance-management-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:191
 - Run thread failed 
java.lang.OutOfMemoryError: Java heap space
2023-06-14 12:45:19,900 [1@group-000200000001-StateMachineUpdater] ERROR o.a.r.s.i.StateMachineUpdater:194
 - 1@group-000200000001-StateMachineUpdater caught a Throwable. 
java.lang.OutOfMemoryError: Java heap space
2023-06-14 12:45:19,899 [pool-6-IoTDB-Flush-2] ERROR o.a.i.d.e.s.TsFileProcessor:1064 
- root.test_machine-20: 
	/opt/apache-iotdb-1.0.1-all-bin/data/datanode/data/sequence/root.test_machine/20/2788/1686721098963-103-0-0.tsfile 
	meet error when flushing a memtable, change system mode to error 
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
2023-06-14 12:45:19,899 [pool-17-IoTDB-Compaction_Schedule-root.omega40h-17-1] ERROR o.a.i.d.e.s.DataRegion:2140
 - Meet error in compaction schedule. 
java.lang.OutOfMemoryError: Java heap space
2023-06-14 12:45:19,899 [pool-16-IoTDB-Compaction_Schedule-root.sg27-4-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:179
 - Schedule task failed 
java.lang.OutOfMemoryError: Java heap space
2023-06-14 12:45:19,909 [DataNodeInternalRPC-Service] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 
- Exception in thread DataNodeInternalRPC-Service-109 
java.lang.OutOfMemoryError: Java heap space
2023-06-14 12:45:19,906 [ForkJoinPool.commonPool-worker-1] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31
 - Exception in thread ForkJoinPool.commonPool-worker-1-9891 
java.lang.OutOfMemoryError: Java heap space
	at java.util.concurrent.ForkJoinPool$WorkQueue.growArray(ForkJoinPool.java:886)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1687)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2023-06-14 12:45:19,915 [pool-6-IoTDB-Flush-2] ERROR o.a.i.c.e.HandleSystemErrorStrategy:37
 - Unrecoverable error occurs! Change system status to read-only because handle_system_error is CHANGE_TO_READ_ONLY. 
   Only query statements are permitted! 
java.lang.RuntimeException: System mode is set to READ_ONLY
	at org.apache.iotdb.commons.enums.HandleSystemErrorStrategy.handle(HandleSystemErrorStrategy.java:37)
	at org.apache.iotdb.commons.conf.CommonConfig.handleUnrecoverableError(CommonConfig.java:282)
	at org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:1069)
	at org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:108)
	at org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-14 12:45:19,907 [qtp1513029997-9878] ERROR o.a.i.d.a.ClusterAuthorityFetcher:217 - Failed to connect to config node. 
2023-06-14 12:45:19,909 [qtp1513029997-9863] ERROR o.a.i.d.a.ClusterAuthorityFetcher:217 - Failed to connect to config node. 
2023-06-14 12:45:19,915 [JvmPauseMonitor0] ERROR o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread JvmPauseMonitor0-94 
java.lang.OutOfMemoryError: Java heap space
2023-06-14 12:45:19,921 [pool-6-IoTDB-Flush-2] ERROR o.a.i.c.conf.CommonConfig:313
 - Change system status to ReadOnly! Only query statements are permitted! 
java.lang.RuntimeException: System mode is set to READ_ONLY
	at org.apache.iotdb.commons.conf.CommonConfig.setNodeStatus(CommonConfig.java:313)
	at org.apache.iotdb.commons.enums.HandleSystemErrorStrategy.handle(HandleSystemErrorStrategy.java:40)
	at org.apache.iotdb.commons.conf.CommonConfig.handleUnrecoverableError(CommonConfig.java:282)
	at org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:1069)
	at org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:108)
	at org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-14 12:45:19,921 [pool-6-IoTDB-Flush-2] ERROR o.a.i.d.e.s.TsFileProcessor:1071 - root.test_machine-20: /opt/apache-iotdb-1.0.1-all-bin/data/datanode/data/sequence/root.test_machine/20/2788/1686721098963-103-0-0.tsfile 
- IOTask meets error, truncate the corrupted data 
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
2023-06-14 13:32:25,471 [pool-51-IoTDB-MPPCoordinator-13$20230614_080225_29492_1.1.0] ERROR o.a.i.d.m.e.e.RegionReadExecutor:56 - Execute FragmentInstance in ConsensusGroup SchemaRegion[1] failed. 
org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
	at org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:321)
	at org.apache.iotdb.db.mpp.execution.executor.RegionReadExecutor.execute(RegionReadExecutor.java:48)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchLocally(FragmentInstanceDispatcherImpl.java:231)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:137)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:102)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 1@group-000200000001 is not in [RUNNING]: current state is CLOSED
	at org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:733)
	at org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:253)
	at org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:732)
	at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:822)
	at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequest(RaftServerImpl.java:944)
	at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequest(RaftServerProxy.java:442)
	at org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:316)
	... 8 common frames omitted
2023-06-14 13:33:11,148 [pool-51-IoTDB-MPPCoordinator-19$20230614_080311_29496_1.1.0] ERROR o.a.i.d.m.e.e.RegionReadExecutor:56 - Execute FragmentInstance in ConsensusGroup SchemaRegion[1] failed. 
org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
	at org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:321)
	at org.apache.iotdb.db.mpp.execution.executor.RegionReadExecutor.execute(RegionReadExecutor.java:48)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchLocally(FragmentInstanceDispatcherImpl.java:231)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:137)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:102)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 1@group-000200000001 is not in [RUNNING]: current state is CLOSED
	at org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:733)
	at org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:253)
	at org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:732)
	at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:822)
	at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequest(RaftServerImpl.java:944)
	at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequest(RaftServerProxy.java:442)
	at org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:316)
	... 8 common frames omitted
2023-06-14 16:12:59,943 [pool-51-IoTDB-MPPCoordinator-10$20230614_104259_29553_1.1.0] ERROR o.a.i.d.m.e.e.RegionReadExecutor:56 - Execute FragmentInstance in ConsensusGroup SchemaRegion[1] failed. 
org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request failed
	at org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:321)
	at org.apache.iotdb.db.mpp.execution.executor.RegionReadExecutor.execute(RegionReadExecutor.java:48)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchLocally(FragmentInstanceDispatcherImpl.java:231)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:137)
	at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:102)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.ratis.protocol.exceptions.ServerNotReadyException: 1@group-000200000001 is not in [RUNNING]: current state is CLOSED
	at org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:733)
	at org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:253)
	at org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:732)
	at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:822)
	at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequest(RaftServerImpl.java:944)
	at org.apache.ratis.server.impl.RaftServerProxy.submitClientRequest(RaftServerProxy.java:442)
	at org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:316)
	... 8 common frames omitted

Thank you

[jixuan1989]

There are many reasons that may cause OOM. Current information is not enough to locate the issue.
However, as far as I know, there is some memory leakage in the query process of v1.0.x.
So, if your workload has many query requests, try to upgrade to v1.1.x, and let's see whether the issue is solved.

[rsupreet]

Hi @jixuan1989,
Thank you for the reply.

if your workload has many query requests,

I am continuously(24/7) inserting some time-series test data to the IoTDB using the REST API service provided by IoTDB. On 8th day I got the above "java.lang.OutOfMemoryError: Java heap space" error.
Also, w.r.t query requests I make query request through Grafana to view the test data collected in IoTDB but it is very minimal.

try to upgrade to v1.1.x, and let's see whether the issue is solved.

Since I am using Grafana version v7.5.17, I was not able to use IoTDB v1.1.x version.
Will try to see if I can upgrade to the IoTDB v1.1.x version with compatible Grafana-Plugin supporting Grafana v7.5.17.