How to avoid huge core dump file under /tmp/tpu_logs/core

[mar-muel]

Hello 👋 I'm facing an issue when running jax workers in production on Cloud TPUs v4-8s. I run out of disk space due to very large core dumps being written after my jax worker crashes (possibly related to OOM). E.g. on one of my VMs I found a file under /tmp/tpu_logs/core that was a 52GB binary file 😱. I'd imagine it also takes quite long for this core dump to be written to disk.

I would like to avoid core dumps being written in production environments for now. Is there an environment variable I could use for this? Or can this be solved by setting --ulimit core=0 when running my jax worker with docker run? Thanks for your help! 🙏

[mar-muel]

Update (in case anyone else runs into this): --ulimit core=0 solved the issue 😄

[hawkinsp]

Yes, I'm not exactly sure what else to suggest other than using a lower core dump size limit, other than "don't crash" ;-)

[mar-muel]

Right - best thing would be to resolve the core issue of course 😅 The error I'm seeing in the logs isn't very insightful to me - it happens after a few hours (while being in an idle state). Any idea what this could be related to?

[mutex.cc : 463] RAW: Lock blocking 0x563c15e64f10   @
[mutex.cc : 1957] RAW: Check (v & (kMuWait | kMuWrWait)) != kMuWrWait failed: Lock: Mutex corrupt: waiting writer with no waiters: 0x563c06016f30
https://symbolize.stripped_domain/r/?trace=7fb2e051dccc,7fb2e04cef8f,563c5f2bcdff&map=
*** SIGABRT received by PID 7 (TID 769292) on cpu 78 from PID 7; stack trace: ***
PC: @     0x7fb2e051dccc  (unknown)  (unknown)
    @     0x7fb17f2e6c9a       1152  (unknown)
    @     0x7fb2e04cef90       9680  (unknown)
    @     0x563c5f2bce00  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7fb2e051dccc,7fb17f2e6c99,7fb2e04cef8f,563c5f2bcdff&map=4e45faae3f4019f2630b86369405b370:7fb171e00000-7fb17f4f6f30
E0908 03:50:33.306632  769292 coredump_hook.cc:447] RAW: Remote crash data gathering hook invoked.
E0908 03:50:33.306647  769292 coredump_hook.cc:486] RAW: Skipping coredump since rlimit was 0 at process start.
E0908 03:50:33.306657  769292 client.cc:270] RAW: Coroner client retries enabled (b/136286901), will retry for up to 30 sec.
E0908 03:50:33.306659  769292 coredump_hook.cc:542] RAW: Sending fingerprint to remote end.
E0908 03:50:33.306678  769292 coredump_hook.cc:551] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] stat failed on crash reporting socket /var/google/services/logmanagerd/remote_coredump.socket (Is the listener running?): No such file or directory
E0908 03:50:33.306681  769292 coredump_hook.cc:603] RAW: Dumping core locally.
E0908 03:50:33.817341  769292 process_state.cc:783] RAW: Raising signal 6 with default behavior

[hawkinsp]

Well, I'd guess memory corruption is likely, given the absl::Mutex that is reporting that error is part of extremely well-tested code.

Now, that could occur because of either a bug in the Cloud TPU plugin, or it could occur because of bugs in any other C/C++ code linked into the same binary. It's hard to say which without a reproduction. If you need us to look at this, you'd probably need to provide instructions to reproduce either here or via Cloud TPU support channels.

[mar-muel]

Yeah it's possible we're having a RAM memory spike that brings the worker down. I've been fighting with memory leaks quite a bit when copying data between TPU (jax) and host (numpy). Unfortunately very hard to reproduce. Thanks anyway for your help!