My server randomly hangs, with dumpexceptions showing System.ExecutionEngineException

[luukvnes]

Hi,

My asp.net server randomly stops working, while the process is still active and no exception is thrown. All threads "pause" and nothing happens anymore. As far as I can tell using dotnet-dump there is no deadlock, the syncblk table shows every line with Monitorheld 0. A strange thing i found is that using dumpexceptions I can see a System.ExecutionEngineException, but this does not contain any information about why or where this happened. I am not really sure where to go next to find the issue. If anyone could help me with finding the problem that would be amazing.

---

   1 System.Threading.Monitor.Wait(Object, Int32)
   1 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
   1 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, CancellationToken)
   1 System.Threading.Tasks.Task.InternalWaitCore(Int32, CancellationToken)
   1 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions)
   1 Program.<Main>(String[])


________________________________________________
~~~~ 143232
   1 Interop+Sys.Read(SafeHandle, Byte*, Int32)
   1 System.IO.FileSystemWatcher+RunningInstance.TryReadEvent(NotifyEvent ByRef)
   1 System.IO.FileSystemWatcher+RunningInstance.ProcessEvents()


________________________________________________
~~~~ 1432ce
   1 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
   1 System.Threading.ManualResetEventSlim.Wait(TimeSpan)
   1 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop()
   1 System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)


________________________________________________
     ~~~~ 143234
        1 System.Threading.Monitor.Wait(Object, Int32)
        1 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
        1 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, CancellationToken)
        1 System.Threading.Tasks.Task.InternalWaitCore(Int32, CancellationToken)
        1 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions)
        1 Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost)
        1 Program+<<Main>$>d__0.MoveNext()
        1 System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
        --------------- **Some startup code, removed here for clarity (running the program without this code does not help)** --------------------
        1 System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, Memory<Byte>, SocketFlags, SocketError)
        1 System.Net.Sockets.SocketAsyncEngine.Execute()
        1 System.Threading.ThreadPoolWorkQueue.Dispatch()
     ~~~~ 55b8b
        1 System.Threading.LowLevelLifoSemaphore.WaitNative(SafeWaitHandle, Int32)
        1 System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32)
        1 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean)
   2 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()


________________________________________________
~~~~ 143235
   1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
   1 System.Threading.PortableThreadPool+GateThread.GateThreadStart()


________________________________________________
~~~~ 143268
   1 System.Net.Sockets.SocketAsyncEngine.EventLoop()


________________________________________________
~~~~ 14326a
   1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
   1 System.Threading.TimerQueue.TimerThread()


________________________________________________
~~~~ 1432b0
   1 System.Threading.Thread.Sleep(Int32)
   1 Program+<>c.<<Main>$>b__0_12()
   
 dumpexceptions                                                                                                                                                                                                                                                                                       
      Address      MethodTable Message Name
 7ad546404a48     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad546496b38     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad5465b37a8     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad57c000088     7b155c1ce440 System.OutOfMemoryException
     Message: <null>
 7ad57c000100     7b155c1ce568 System.StackOverflowException
     Message: <null>
 7ad57c000178     7b155c1ce690 System.ExecutionEngineException
     Message: <null>
 7ad57c36b5d8     7b155e587a08 System.ObjectDisposedException
     Message: Cannot access a disposed object.
 7ad546c20d70     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad546d5dfc0     7b15639527f0 Microsoft.AspNetCore.Connections.ConnectionAbortedException
     Message: The Socket transport's send loop completed gracefully.
 7ad57e5d9280     7b155df682e0 System.ArgumentNullException
     Message: Value cannot be null.

   Total: 10 objects
   
 bottom part of dumpheap
 7b155e4e4ea0  2,545    386,840 System.Reflection.Emit.DynamicILGenerator
7b1561a850b0  1,546    494,720 System.Collections.Concurrent.ConcurrentDictionary<System.String, Microsoft.EntityFrameworkCore.Infrastructure.Annotation>+VolatileNode[]
7b155e4f4720  5,889    518,232 System.Reflection.RuntimeParameterInfo
7b155c1c8d38  7,710    731,324 System.Int32[]
7b155e4f1a40  7,456    775,424 System.Reflection.RuntimeMethodInfo
7b155c277d30  1,702    939,938 System.Char[]
7b155c172278  6,388  1,078,648 System.Object[]
7b155b234910 64,271  1,542,504 System.Object
7b155c1cbe30 44,794  6,382,110 System.String
7b155e3a9240 14,496  8,133,504 System.Byte[]
5633de175bd0  3,884 13,070,488 Free
Total 386,593 objects, 48,672,038 bytes

Fragmented blocks larger than 0.5 MB:
        Address           Size      Followed By
   7ad54be45858        800,048     7ad54bf08d88 System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder<Npgsql.IBackendMessage>+StateMachineBox<Npgsql.Internal.NpgsqlConnector+<ReadMessageLong>d__235>
   7ad57fdad190        935,600     7ad57fe91840 System.Buffers.SharedArrayPoolPartitions

[gfoidl]

System.ExecutionEngineException

Please open an issue in https://github.com/dotnet/runtime (with the same content as here, I'm not able to transfer the discussion here to the issue there). Ideally you could attach / link the dump file -- when it's private one of the team there will ask you to send it privately, etc. for further investigation.

[gfoidl]

Thanks for creating the issue: dotnet/runtime#117496