Deadlock in GC

[jhudsoncedaron]

Description

Active worker thread is hung up with a diagnostic call stack that looks something like this:

ntdll.dll!syscall_gate
KERNELBASE.dll!WaitForMultipleObjects
???
CORINFO_HELP_NEWARR_1_VC
inlined!StringBuilder.ctor
Cedaron.Common.HL7.DLL!Cedaron.Common.HL7.HL7Segment.HL7Decode

All managed stacks as follows:

Main thread:
 	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	[Managed to Native Transition]	
 	System.Private.CoreLib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) Line 156	C#
 	System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 561	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3072	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3007	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2768	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout) Line 2728	C#
 	[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]	
	System.IO.FileSystem.Watcher.dll!System.IO.FileSystemWatcher.WaitForChanged(System.IO.WatcherChangeTypes changeType, int timeout) Line 631	C#
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Program.Main(string[] args) Line 300	C#

Background task scheduler thread:
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.Win32WaitForAnyObject(nint[] waithandle, int tv_sec, int tv_msec)	Unknown
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.WaitForAnyRead(System.Collections.Generic.IList<Cedaron.Common.Either<System.Net.Sockets.Socket, Cedaron.Common.Portable.IMultiWaitable>> waitfor, int tv_sec, int tv_msec)	Unknown
	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.InterfaceEngine.BackgroundJobRunner()

Background keepalive thread:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
	System.Private.CoreLib.dll!System.Threading.Thread.Sleep(int millisecondsTimeout) Line 367	C#
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.APIConfiguration.RefreshToken()	Unknown

.NET TP Gate:
 	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.WaitHandle.WaitOneNoCheck(int millisecondsTimeout) Line 128	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.GateThread.GateThreadStart() Line 48	C#

.NET ThreadPool IO:
 	ntdll.dll!00007ff9df4030c4()	Unknown
 	KERNELBASE.dll!00007ff9dcab2724()	Unknown
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.IOCompletionPoller.Poll() Line 188	C#

.NET Timer:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.WaitHandle.WaitOneNoCheck(int millisecondsTimeout) Line 128	C#
 	System.Private.CoreLib.dll!System.Threading.TimerQueue.TimerThread() Line 87	C#

Connection Listener:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.Win32WaitForAnyObject(nint[] waithandle, int tv_sec, int tv_msec)	Unknown
 	Cedaron.Common.dll!Cedaron.Common.Portable.MultiWaiter.WaitForAnyRead(System.Collections.Generic.IList<Cedaron.Common.Either<System.Net.Sockets.Socket, Cedaron.Common.Portable.IMultiWaitable>> waitfor, int tv_sec, int tv_msec)	Unknown
 	Cedaron.HL7.Client.dll!Cedaron.HL7.Client.SocketTransport.GetConnected(System.Action monitor, int monitorinterval)	Unknown
>	Cedaron.Interface.Engine.dll!Cedaron.Interface.Receivers.TCP.Cedaron.Interface.Declarations.IInterfaceReceiver.ReceiveLoop()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Interface.ReceiveLoop()

.NET Finalizer:
>	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

Only Active Worker Thread:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	Cedaron.Common.HL7.dll!Cedaron.Common.HL7.HL7Segment.HL7Decode(string data, int offset, int length, char msh1, string msh2, bool translateunknownesc) Line 391	C#
 	Cedaron.Common.HL7.dll!Cedaron.Common.HL7.HL7Segment.HL7Segment(string msgdata, int offset, int length, char msh1, string msh2, bool translateunknownesc) Line 300	C#
 	Cedaron.Common.HL7.dll!Cedaron.Common.HL7.HL7Message.HL7Message(byte[] message, int offset, int length, bool translateunknownesc) Line 523	C#
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.ADT.UpdateManualPromoteRecords()
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.ADT.ProcessADT()
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.ADT.Cedaron.Interface.Declarations.IMessageProcessor.ProcessMessage()
 	Cedaron.Interface.FormBuilder.dll!Cedaron.Interface.FormBuilder.DatabaseArchiver.Cedaron.Interface.Declarations.IMessageProcessor.ProcessMessage()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Interface.ProcessMessage()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Receivers.TCP.Cedaron.Interface.Declarations.IInterfaceReceiver.ReceiveLoop.AnonymousMethod__0(Cedaron.HL7.Client.HL7BlockingStream stream)
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Receivers.TCP.Cedaron.Interface.Declarations.IInterfaceReceiver.ReceiveLoop()
 	Cedaron.Interface.Engine.dll!Cedaron.Interface.Engine.Interface.ReceiveLoop()

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Debugger:
>	ntdll.dll!00007ff9df400d24()	Unknown
 	KERNELBASE.dll!00007ff9dc9d464c()	WaitForSingleObject
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Event Pipe:
 	[Waiting on a lock, load symbols for ntdll.dll to show thread lock information]	
>	ntdll.dll!00007ff9df403cc4()	Unknown
 	ntdll.dll!00007ff9df379f09()	Unknown
 	ntdll.dll!00007ff9df3774c2()	Unknown
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET Server GC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET TP Worker:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int timeoutMs) Line 140	C#
 	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.Wait(int timeoutMs, bool spinWait) Line 65	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 100	C#

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET BGC:
>	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

.NET TP Worker:
 	ntdll.dll!00007ff9df400254()	Unknown
 	KERNELBASE.dll!00007ff9dc9ad71e()	WaitForMultipleObjects
 	[Managed to Native Transition]	
>	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int timeoutMs) Line 140	C#
 	System.Private.CoreLib.dll!System.Threading.LowLevelLifoSemaphore.Wait(int timeoutMs, bool spinWait) Line 65	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 100	C#
 	[Native to Managed Transition]	
 	kernel32.dll!00007ff9dde74cb0()	Unknown
 	ntdll.dll!00007ff9df3deceb()	Unknown

Reproduction Steps

No controlled reproduction available. We currently get the issue about three times per two weeks.

Expected behavior

new StringBuilder() does not hang forever.

Actual behavior

new StringBuilder() calls WaitForMultipleObjects, which does not return

Regression?

Yes; Issue is less than three months old.

Known Workarounds

No response

Configuration

.NET Runtime 8.0.7 via dotnet publish -r win-x64
OS: Windows Server Core; probably 2022

Other information

We have a full memory dump but can't release it publicly. This is a production instance and PHI is on the stack.

[jkotas]

Is this with regular CoreCLR with a JIT or with native AOT? RhpNewArray is a native AOT specific method, but the stacktraces you have shared look like regular CoreCLR with a JIT.

[jhudsoncedaron]

@jkotas : I'm sorry I don't understand the difference. When I did a code search for CORINFO_HELP_NEWARR_1_VC ; it seems to be mapped to RhpNewArray by RyuJIT. The environment is whatever dotnet publish -r win-x64 spits out.

[jkotas]

cc @VSadov

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[mangod9]

Hi @jhudsoncedaron, you mention that this started occurring recently, so do you observe that this is a regression in 8.0.7?

[jhudsoncedaron]

@mangod9 : Barring exotic hypotheses, yes; however we jumped straight from 8.0.5 to 8.0.7 and are not sure of the exact start date of the problem. Support was slow to report the issue to engineering. In addition, we just turned on gcDynamic a couple of months ago.

However exotic hypotheses are on the table; the data being processed as seen in the dump is trying to allocate about 5,000 StringBuilder objects in 50ms.

[mangod9]

Hey @jhudsoncedaron, are you able to privately share a dump to investigate further?

[jhudsoncedaron]

@mangod9 : I will ask the guy capable of approving immediately. If that doesn't work, there's other ways of getting necessary information from a dump.

[jhudsoncedaron]

@mangod9 : Dump upload is not happening. So it comes down to tell me what to look for and I can look for it.

[jhudsoncedaron]

So I feel the need to explain a few things.

We are not sitting idle on this. At once other things are happening.

1. We are trying to switch that one instance to workstation GC to see if it clears it (this will hit production first).
2. We are optimizing the allocation-hot loop (in fact the very one with the blocked thread) to allocate less stuff. This should make it happen less often.
3. We looked at the wait handle code; and found a very unlikey pathway that could corrupt wait handle references. This requires two billion exceptions to be thrown from the same finally block where the only possible exception is OutOfMemoryException allocating a List enumerator. On triggering the fault it would overflow the reference counter inside SafeWaitHandle; but I don't see how it could cause this particular fault. (Fix: replace that particular foreach loop with a counted loop).
4. We are trying to find a way to reproduce this in a dev environment. It's not looking promising.
5. The dump contains Medical Records data. There's no way any production dump is getting uploaded anywhere until all other options are exhausted. I am reasonably certain a paid support case will happen first.