Deadlock in php-fpm

[marksteward]

This might be a race triggered by constant OOMing. All workers look like this for hours, and are processing no requests:

(gdb) bt
#0  futex_abstimed_wait (private=128, abstime=0x0, clockid=0, expected=3, futex_word=<optimized out>) at ../sysdeps/nptl/futex-internal.h:287
#1  __pthread_rwlock_rdlock_full (abstime=0x0, clockid=0, rwlock=0x7f4dccf2a078) at pthread_rwlock_common.c:460
#2  __GI___pthread_rwlock_rdlock (rwlock=0x7f4dccf2a078) at pthread_rwlock_rdlock.c:27
#3  0x00007f4dd06af980 in apc_lock_rlock_impl (lock=<optimized out>) at /tmp/pear/temp/apcu/apc_lock.c:98
#4  apc_lock_rlock (lock=<optimized out>) at /tmp/pear/temp/apcu/apc_lock.c:333
#5  0x00007f4dd06b2a6d in apc_cache_rlock (cache=<optimized out>, cache=<optimized out>) at /tmp/pear/temp/apcu/apc_cache.h:291
#6  apc_cache_rlock (cache=0x557742178260) at /tmp/pear/temp/apcu/apc_cache.h:289
#7  apc_cache_fetch (cache=0x557742178260, key=0x7f4dd04d2ba0, t=t@entry=1758890063, dst=dst@entry=0x7f4dd0414f10) at /tmp/pear/temp/apcu/apc_cache.c:857
#8  0x00007f4dd06b04be in zif_apcu_fetch (execute_data=0x7f4dd0414fa0, return_value=0x7f4dd0414f10) at /tmp/pear/temp/apcu/php_apc.c:631
#9  0x0000557732b8c654 in execute_ex ()
#10 0x0000557732b9213d in zend_execute ()
#11 0x0000557732b0bdd4 in zend_execute_scripts ()
#12 0x0000557732aab1e0 in php_execute_script ()
#13 0x0000557732844161 in ?? ()
#14 0x00007f4dd30c0d0a in _IO_acquire_lock_fct (p=<synthetic pointer>) at libioP.h:885
#15 getwchar () at getwchar.c:36
#16 0x00007f4dd3c34180 in ?? ()
#17 0x0000000000000000 in ?? ()

Has happened 3 times now.

[marksteward]

Looks like krakjoe/apcu#582 (although we're on PHP 7.4 still).

For now I've commented out apc.enabled and apcu.enabled in /usr/local/etc/php/conf.d, set $wgMainCacheType = CACHE_DB; in the LocalSettings.php, and bounced php-fpm.

We probably want to avoid apcu, "The symptoms would also be consistent with a process crashing while holding the lock" isn't great.

[madmajestro]

What exactly do you mean by OOming? If that means that the OOM killer occasionally kills fpm-processes, then that's probably the problem. APCu uses POSIX rwlocks, which don't support robustness. This means that deadlocks can occur if the OOM killer kills a fpm-process holding a lock.

That wouldn't be a new problem and it would not be related to krakjoe/apcu#582. It exists for a long time and for many versions. See krakjoe/apcu#486.

[marksteward]

Yeah, I suspect that's the case. I found the more recent one from a search for apc_lock_rlock (rather than apc_lock_wlock).

It's a shame it's not possible to let each worker look after its own apc cache, because processes can die in any environment.

[madmajestro]

But you can't control which fpm process serves a request, which means that each process woud need to build cache entries for all possible clients. That would not make sense performancewise. In Addition it would waste memory.

And speaking about memory:
If you see the oom killer regularly. You should try to find a way to avoid this. It can cause problems in many areas.