RWIs fill out the whole memory space for YaCy

[okybaca]

When YaCy instance crawls a lot, RWI kelondro index becomes too large, and
YaCy gets stucked with out of memory exception. Instance gets blocked and
doesn't work any more. Only killing the instance, manual deleting the
excessive RWI and restart helps.

RWI transfer and 'dissolving' is slow (as described in #724) and crawling
easily overfills RWI index.

Possible solution could be:

• when certain memory limit is reached, the oldest RWIs are deleted
  automatically, similarly to DISK LIMITS

• employ more effective RWIs memory behavior (i got no idea, but i guess
  that all rwi's indexes are kept in memory (?)), so that the bigger index
  doesn't consume more and more RAM, above the limits

• use modern garbage collector, like described here:
  https://www.geeksforgeeks.org/java/demystifying-memory-management-in-java/
  that would probably help also in some other aspects of YaCy operation

Reported also in the forum:
https://community.searchlab.eu/t/topic/1088

log:

...
W 2025/09/03 12:00:20 ConcurrentLog * net.yacy.cora.util.SpaceExceededException: 25846200 bytes needed for RowCollection grow after OutOfMemoryError J
ava heap space: 72781488 free at Wed Sep 03 12:00:20 CEST 2025 
net.yacy.cora.util.SpaceExceededException: 25846200 bytes needed for RowCollection grow after OutOfMemoryError Java heap space: 72781488 free at Wed S
ep 03 12:00:20 CEST 2025 
        at net.yacy.kelondro.index.RowCollection.ensureSize(RowCollection.java:276) 
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:425) 
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:403) 
        at net.yacy.kelondro.index.RAMIndex.addUnique(RAMIndex.java:216) 
        at net.yacy.kelondro.index.RAMIndexCluster.addUnique(RAMIndexCluster.java:133) 
        at net.yacy.kelondro.index.RowHandleMap.<init>(RowHandleMap.java:116) 
        at net.yacy.kelondro.blob.HeapReader.initIndexReadDump(HeapReader.java:179) 
        at net.yacy.kelondro.blob.HeapReader.<init>(HeapReader.java:91) 
        at net.yacy.kelondro.blob.HeapModifier.<init>(HeapModifier.java:58) 
        at net.yacy.kelondro.blob.ArrayStack.<init>(ArrayStack.java:209) 
        at net.yacy.kelondro.rwi.ReferenceContainerArray.<init>(ReferenceContainerArray.java:68) 
        at net.yacy.kelondro.rwi.IndexCell.<init>(IndexCell.java:99) 
        at net.yacy.search.index.Segment.connectRWI(Segment.java:162) 
        at net.yacy.search.Switchboard.<init>(Switchboard.java:610) 
        at net.yacy.yacy.startup(yacy.java:212) 
        at net.yacy.yacy.main(yacy.java:809) 
I 2025/09/03 12:00:20 KELONDRO * HeapReader: generating index for [REDACTED]/DATA/INDEX/freeworld/SEGMENTS/default/text.index.202508
20035645330.blob, 6624 MB. Please wait. 
W 2025/09/03 12:01:17 ConcurrentLog * net.yacy.cora.util.SpaceExceededException: 9419140 bytes needed for RowCollection grow after OutOfMemoryError Ja
va heap space: 35786272 free at Wed Sep 03 12:01:17 CEST 2025 
net.yacy.cora.util.SpaceExceededException: 9419140 bytes needed for RowCollection grow after OutOfMemoryError Java heap space: 35786272 free at Wed Se
p 03 12:01:17 CEST 2025 
        at net.yacy.kelondro.index.RowCollection.ensureSize(RowCollection.java:276) 
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:425) 
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:403) 
        at net.yacy.kelondro.index.RAMIndex.addUnique(RAMIndex.java:216) 
        at net.yacy.kelondro.index.RAMIndexCluster.addUnique(RAMIndexCluster.java:133) 
        at net.yacy.kelondro.index.RowHandleMap.putUnique(RowHandleMap.java:294) 
        at net.yacy.kelondro.index.RowHandleMap$initDataConsumer.call(RowHandleMap.java:499) 
        at net.yacy.kelondro.index.RowHandleMap$initDataConsumer.call(RowHandleMap.java:438) 
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) 
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 
        at java.base/java.lang.Thread.run(Thread.java:1589) 
I 2025/09/03 12:01:34 ConcurrentLog shutdown of ConcurrentLog.Worker void because it was not running. 
E 2025/09/03 12:01:34 UNCAUGHT-EXCEPTION * Thread main: Java heap space 
java.lang.OutOfMemoryError: Java heap space 

java.lang.OutOfMemoryError: Java heap space 
E 2025/09/03 12:01:34 ConcurrentLog Java heap space 
java.lang.OutOfMemoryError: Java heap space 

        at net.yacy.kelondro.blob.HeapModifier.<init>(HeapModifier.java:58) 
        at net.yacy.kelondro.blob.ArrayStack.<init>(ArrayStack.java:209) 
        at net.yacy.kelondro.rwi.ReferenceContainerArray.<init>(ReferenceContainerArray.java:68) 
        at net.yacy.kelondro.rwi.IndexCell.<init>(IndexCell.java:99) 
        at net.yacy.search.index.Segment.connectRWI(Segment.java:162) 
        at net.yacy.search.Switchboard.<init>(Switchboard.java:610) 
        at net.yacy.yacy.startup(yacy.java:212) 
        at net.yacy.yacy.main(yacy.java:809) 
I 2025/09/03 12:00:20 KELONDRO * HeapReader: generating index for [REDACTED]/DATA/INDEX/freeworld/SEGMENTS/default/te
20035645330.blob, 6624 MB. Please wait. 
W 2025/09/03 12:01:17 ConcurrentLog * net.yacy.cora.util.SpaceExceededException: 9419140 bytes needed for RowCollection grow after OutO
va heap space: 35786272 free at Wed Sep 03 12:01:17 CEST 2025 
net.yacy.cora.util.SpaceExceededException: 9419140 bytes needed for RowCollection grow after OutOfMemoryError Java heap space: 35786272
p 03 12:01:17 CEST 2025 
        at net.yacy.kelondro.index.RowCollection.ensureSize(RowCollection.java:276) 
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:425) 
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:403) 
        at net.yacy.kelondro.index.RAMIndex.addUnique(RAMIndex.java:216) 
        at net.yacy.kelondro.index.RAMIndexCluster.addUnique(RAMIndexCluster.java:133) 
        at net.yacy.kelondro.index.RowHandleMap.putUnique(RowHandleMap.java:294) 
        at net.yacy.kelondro.index.RowHandleMap$initDataConsumer.call(RowHandleMap.java:499) 
        at net.yacy.kelondro.index.RowHandleMap$initDataConsumer.call(RowHandleMap.java:438) 
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) 
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) 
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) 
        at java.base/java.lang.Thread.run(Thread.java:1589) 
I 2025/09/03 12:01:34 ConcurrentLog shutdown of ConcurrentLog.Worker void because it was not running. 
E 2025/09/03 12:01:34 UNCAUGHT-EXCEPTION * Thread main: Java heap space 
...

[okybaca]

copilot's suggestions

[smokingwheels]

How much RAM does your system have?
What is the rough size of your index?
What is the version of your java?

ChatGPT 5 suggestions

ChatGPT 4 suggestions

[okybaca]

How much RAM does your system have?

9 GB reserved to yacy.

What is the rough size of your index?

stucks around ~550GB of RWI index

What is the version of your java?

openjdk 19 but i think, it doesn't really matter, it's an error in yacy logic, could happen both with larger index and larger RAM reserved, and smaller index/less RAM. the point is it crashes with index growth - and shouldn't.

[smokingwheels]

I have tested my backup peer and it failed to start with

JAVA_ARGS="-Xmx16g -Xms6g -XX:+UseG1GC -XX:+UseStringDeduplication $JAVA_ARGS";

I increased to -Xmx18g and it started, thus calculated ~11 GB for your peer to run.

Is there any way to increase the memory capacity in your peer?

📊 YaCy Memory Usage Estimate (Based on Index Size)

🧮 Scenario: Estimating RAM for a 550 GB YaCy Index

Parameter | Value -- | -- Current Known Setup | 1.2 TB disk, 47.8M links, 24 GB RAM Target Setup | 550 GB disk (estimated ~22M links) Estimated RAM Required | 10–12 GB RAM

---

📐 Calculation Details

Estimation Based on Disk Space:

(550 GB / 1200 GB) × 24 GB RAM = ~11 GB RAM

Estimation Based on Link Count:

(550 / 1200) × 47.8M links = ~21.9M links
(24 / 47.8) × 21.9 = ~11 GB RAM

[okybaca]

Is there any way to increase the memory capacity in your peer?

Yes, I could do that, did many times, in fact, but that's not the point. The point is that, with intensive crawling, RAM usage by RWI grows over time until it becomes too much and the peer crashes, requiring manual involvement. That's the bug, I described and believe it should be solved in the code itself. I tried using G1GC already (as a startup parameter for Java `-XX:+UseG1GC` and didn't help).