queues_terminate_all= Fails to clear crawler stack

[smokingwheels]

Windows 11 256 GB ram yacy 150 gb

Result files leftover in queues folder.

I 2025/05/08 19:31:35 Crawl Start * queues_terminate_all=
I 2025/05/08 19:31:35 NoticedURL * CLEARING ALL STACKS
I 2025/05/08 19:31:35 FileUtils * deletedelete:
E 2025/05/08 19:31:35 FileUtils * cannot delete file G:\yacy_search_server\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\fr.linkedin.com-#OAtAY4.443\0003.stack
I 2025/05/08 19:31:35 SWITCHBOARD * Excluded 0 words in URL

Also terminate all hang.

E 2025/05/08 19:32:24 FileUtils * cannot delete file G:\yacy_search_server\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\feedproxy.google.com-#Iu6PtY.80\0002.stack
I 2025/05/08 19:32:24 HostQueue * closing HostQueue, closing 1 depthStacks for host www.biofueldaily.com
I 2025/05/08 19:32:24 FileUtils * deletedelete: G:\yacy_search_server\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\feedproxy.google.com-#Iu6PtY.80\0003.stack

E 2025/05/08 19:32:24 FileUtils * cannot delete file G:\yacy_search_server\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\feedproxy.google.com-#Iu6PtY.80\0003.stack
I 2025/05/08 19:32:24 HostQueue * closing HostQueue, closing 1 depthStacks for host www.institut-agro.fr
I 2025/05/08 19:32:24 FileUtils * deletedelete: G:\yacy_search_server\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\feedproxy.google.com-#Iu6PtY.80\0002.stack

E 2025/05/08 19:32:24 FileUtils * cannot delete file G:\yacy_search_server\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\feedproxy.google.com-#Iu6PtY.80\0002.stack
I 2025/05/08 19:32:24 HostQueue * closing HostQueue, closing 1 depthStacks for host www.chu-nimes.fr
I 2025/05/08 19:32:24 FileUtils * deletedelete: G:\yacy_search_server\DATA\INDEX\freeworld\QUEUES\CrawlerCoreStacks\feedproxy.google.com-#Iu6PtY.80\0003.stack

at shutdown
I 2025/05/08 19:33:57 KELONDRO * HeapReader: wrote a dump for the 9 index entries of stats.bheap in 0 milliseconds.
I 2025/05/08 19:33:57 KELONDRO * HeapReader: close HeapFile stats.bheap
I 2025/05/08 19:33:57 KELONDRO * Heap: closing heap G:\yacy_search_server\DATA\WORK\robots.bheap
I 2025/05/08 19:33:57 KELONDRO * HeapReader: wrote a dump for the 387 gap entries of robots.bheap in 0 milliseconds.
I 2025/05/08 19:33:57 KELONDRO * HeapReader: wrote a dump for the 32140 index entries of robots.bheap in 32 milliseconds.
I 2025/05/08 19:33:57 KELONDRO * HeapReader: close HeapFile robots.bheap
I 2025/05/08 19:33:57 KELONDRO * Heap: closing heap G:\yacy_search_server\DATA\WORK\api.bheap
I 2025/05/08 19:33:57 KELONDRO * HeapReader: wrote a dump for the 0 gap entries of api.bheap in 0 milliseconds.
I 2025/05/08 19:33:57 KELONDRO * HeapReader: wrote a dump for the 9 index entries of api.bheap in 15 milliseconds.
I 2025/05/08 19:33:57 KELONDRO * HeapReader: close HeapFile api.bheap
I 2025/05/08 19:33:58 UPNP * unmapped port 49168 on device Archer_VR2800
W 2025/05/08 19:33:58 ConcurrentLog * java.lang.InterruptedException
java.lang.InterruptedException
at java.base/java.util.concurrent.locks.ReentrantLock$Sync.lockInterruptibly(ReentrantLock.java:159)
at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:372)
at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:430)
at net.yacy.crawler.data.CrawlQueues$Loader.run(CrawlQueues.java:739)
W 2025/05/08 19:33:58 ConcurrentLog * java.lang.InterruptedException
java.lang.InterruptedException
at java.base/java.util.concurrent.locks.ReentrantLock$Sync.lockInterruptibly(ReentrantLock.java:159)
at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:372)
at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:430)
at net.yacy.crawler.data.CrawlQueues$Loader.run(CrawlQueues.java:739)
W 2025/05/08 19:33:58 ConcurrentLog * java.lang.InterruptedException
java.lang.InterruptedException
at java.base/java.util.concurrent.locks.ReentrantLock$Sync.lockInterruptibly(ReentrantLock.java:159)
at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:372)
at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:430)
at net.yacy.crawler.data.CrawlQueues$Loader.run(CrawlQueues.java:739)
W 2025/05/08 19:33:58 ConcurrentLog * java.lang.InterruptedException
java.lang.InterruptedException
at java.base/java.util.concurrent.locks.ReentrantLock$Sync.lockInterruptibly(ReentrantLock.java:159)
at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:372)
at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:430)
at net.yacy.crawler.data.CrawlQueues$Loader.run(CrawlQueues.java:739)
I 2025/05/08 19:33:58 ConcurrentLog shutdown of ConcurrentLog.Worker void because it was not running.
W 2025/05/08 19:33:58 ConcurrentLog * java.lang.InterruptedException

[okybaca]

hi,

thank you for many of yours reports here, as well as in the forum!
it looks like you like stress testing and trying to boost the performance!

there are not many developers around this time (i'm not, definitely).

if you would like to ease the way for anyone who would try to fix that, can you dig into the bug a bit more and find a cause?
are you able to find the exact condition, under which error occurs?

every error report should be replicable, which means, anyone could simulate the conditions and get the same error.
it should contain also what you expected to happen and what happened instead.

next step (if you're guru) could be also digging into the source code and try to find a place, which produces an errror.

[smokingwheels]

Hi,
Thank you @okybaca for the comments.

I had a go at source code the result is here https://github.com/smokingwheels/yacy_search_server_universal.
I'm a just learner when it comes to java.

Windows has troubles with 450000 files and 400000 folders

Yep — that’s a known limitation in Windows. When you start hitting hundreds of thousands of files and folders, the NTFS file system and the Windows Explorer UI itself begin to struggle. Here's why, and what you can do about it:

💥 Why Windows Struggles with 450,000 Files / 400,000 Folders
NTFS Metadata Bloat
Every file/folder adds metadata entries. High volume causes slow reads, directory listings, and backups.

Explorer Indexing and UI Lag
Windows Explorer tries to enumerate and sometimes preview every item, which kills responsiveness.

Antivirus & Defender Scanning
Real-time scanning chokes performance with so many items.

Filesystem Journal and USN Change Journal
The NTFS journaling system gets overwhelmed, slowing down all file operations.

I am testing nextcloud syncing to 450k files from yacy's crawler que and its been recommended to only 10 000 per folder.

[okybaca]

so it seems to be a windows issue?

[smokingwheels]

I think so a windows issue, I had in the order of 454k files and 400k folders in the CrawlerCoreStacks and Explorer was very slow to respond at least 3 min to list folder contents.

A Ctrl A operation never finished.

I manually split CrawlerCoreStacks folder into 15k to 25k in subfolders and proceeded to sync with a Nextcloud server, it took 4 days to complete.