bsod caused by msquic

[MattPOlson]

Hello, We have a windows server 2022 standard that we our using as a node in our k8s cluster. We are experiencing a system crash periodically and the stack trace points to an issues with msquic, I was just wondering if anyone has seen this before or knows what the problem could be.

fffffe09`d6f77720 fffff801`45ec5e4c     : 00000000`00000103 00000000`00000010 ffffe08c`1a8ca010 00000000`00000000 : ksecdd!SspiDeleteSecurityContextAsync+0x99
fffffe09`d6f77780 fffff801`45ec5dcb     : ffffe08c`1a8ca010 00000000`00000000 00000000`00000000 fffff801`3fc42239 : msquic!QuicTlsResetSchannel+0x44
fffffe09`d6f777b0 fffff801`45ec5ac0     : ffffe08c`1c8076e0 ffffe08c`227f14d0 ffffe08c`20c366f8 ffffe08c`229ead50 : msquic!QuicTlsUninitialize+0x37
fffffe09`d6f77870 fffff801`45ec48ff     : ffffe08c`20c36010 ffffe08c`20c36010 00000000`00000000 ffffe08c`222fb580 : msquic!QuicCryptoUninitialize+0x118
fffffe09`d6f778b0 fffff801`45ebabcc     : ffffe08c`20c363a0 fffff801`45eb0000 ffff7a27`00000000 ffffb680`e5c05500 : msquic!QuicConnUninitialize+0x8b
fffffe09`d6f778e0 fffff801`45ec66f6     : ffffe08c`20c36010 ffffffff`ffffffff 00000000`00001301 0624dd2f`1a9fbe77 : msquic!QuicConnDrainOperations+0x58c
fffffe09`d6f779e0 fffff801`45ec64be     : 00000000`00000000 ffffe08c`21d1b6c0 ffffe08c`21d1b6c0 ffffe08c`00000000 : msquic!QuicWorkerProcessConnection+0x126
fffffe09`d6f77ae0 fffff801`3f4e83d5     : ffffe08c`1bbd9040 00000000`00000080 fffff801`45ec5ff0 ffffe08c`21d1b6c0 : msquic!QuicWorkerThread+0x4ce
fffffe09`d6f77bf0 fffff801`3f61aa38     : fffff801`3cc16180 ffffe08c`1bbd9040 fffff801`3f4e8380 00000000`00000246 : nt!PspSystemThreadStartup+0x55
fffffe09`d6f77c40 00000000`00000000     : fffffe09`d6f78000 fffffe09`d6f72000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x28

EXCEPTION_RECORD:  fffffe09d6f774e8 -- (.exr 0xfffffe09d6f774e8)
ExceptionAddress: fffff80142a7c6e9 (ksecdd!SspiDeleteSecurityContextAsync+0x0000000000000099)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000000008
Attempt to read from address 0000000000000008

[nibanks]

For any crash that could be a security issue (not sure if this is one or not), could you please follow the steps at https://github.com/microsoft/msquic/security/policy#reporting-security-issues. Feel free to cc me, nibanks@microsoft.com with any additional info you have.

But I've never looked at a crash with this signature before. Not sure if it's a msquic issue or ksecdd issue. Will follow up separately.

[MattPOlson]

We finally were able to track down what was causing this, the application in the container was trying to reach a file share, and it was failing due to an DNS issues. Windows SMB clients still use TCP by default, but since the first attempt was failing it was then trying to use QUIC, the server it was attempting to connect to isn't configured to use QUIC so it was also failing and causing a BSOD on the server. I was able to replicate the BSOD every time by executing a PowerShell command to create a new smb mapping using QUIC.

New-SmbMapping -LocalPath 'Z:' -RemotePath '\\mydomain.local\Prod\External\Vault\Vol_023' -TransportType QUIC

I would expect it to fail gracefully and not cause the server to BSOD.

[chucker]

I think I just ran into this very issue — ran a build pipeline inside a container, and it tries to access a file share as part of its tasks. Caused a BSoD.

Any idea of a workaround? Can I force the container to never use QUIC?

[nibanks]

The following work around should allow for you to disable SMB's usage of QUIC, machine-wide:

REG.EXE ADD "HKLM\SYSTEM\CurrentControlSet\Services\MRxSmb\Parameters" /f /t REG_DWORD /v HowToUseWskAndQuic /d 1

Note that you should not need to reboot. Also, note that #3248 should fix this issue, but we'll need to backport it (which likely takes a couple months).

[chucker]

Running that on both the host and the container (only the host wasn't enough; perhaps only the container would've been) did the trick. Thanks!

[nibanks]

I think you'd only need to run it on the container.