TCP half-opened socket leak

[troelsarvin]

With GWC 1.26.2, the following is seen:

The longer GWC runs, the more threads are being used, and the more sockets like this is seen in output from "lsof -n -P -p ":

java 487429 username 2290u sock 0,8 0t0 3748864 protocol: TCP

After four days, more than 2000 such sockets exist.

Threads, according to Jolokia:

Memory usage slowly climbs up, too:

Various web pages (such as this one) suggest that such sockets indicate a situation where a socket has been created, but neither connect() nor bind() was performed on it.

The number of such threads is in the same ballpark of threads that look like this in jstack output:

"http-nio-10361-exec-3263" #3458 daemon prio=5 tid=0x00007fb608063c00 nid=0x12e82d waiting on condition [0x00007fb5716fd000]
   java.lang.Thread.State: WAITING (parking)
   JavaThread state: _thread_blocked
 - jdk.internal.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
        - parking to wait for <0x00000000a8314888> (a java/util/concurrent/locks/ReentrantLock$NonfairSync)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=211 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int, boolean, boolean, bo
olean, long) @bci=347, line=715 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=15, line=938 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock$Sync.lock() @bci=9, line=153 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=322 (Compiled frame)
 - org.geowebcache.locks.MemoryLockProvider.getLock(java.lang.String) @bci=45, line=72 (Compiled frame)
 - org.geowebcache.layer.wms.WMSLayer.getMetatilingReponse(org.geowebcache.conveyor.ConveyorTile, boolean) @bci=121, line=337 (Compiled frame)
 - org.geowebcache.layer.wms.WMSLayer.getTile(org.geowebcache.conveyor.ConveyorTile) @bci=131, line=272 (Compiled frame)
 - org.geowebcache.util.ResponseUtils.writeTile(org.geowebcache.filter.security.SecurityDispatcher, org.geowebcache.conveyor.Conveyor, java.lang.Strin
g, org.geowebcache.layer.TileLayerDispatcher, org.geowebcache.storage.DefaultStorageFinder, org.geowebcache.stats.RuntimeStats) @bci=30, line=99 (Comp
iled frame)
...

Here's an example of a request which is served by the GWC:
https://GWCHOSTNAME/service/wmts?layer=theme-xxx&style=&tilematrixset=EPSG%3A4326&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=EPSG%3A4326%3A20&TileCol=1121842&TileRow=199821

I don't know when this started. We have had other GWC installations (which have been updated a number of times) for a while, but those get restarted every night, so we didn't notice the thread build-up.

This situation sounds a bit like #1308 , except

• basic auth is not involved in my case
• I have yet to see GWC stop serving requests due to this (but I suppose things will start to go wrong at some point)