RequestClose: ObjectdisposedExceptions in Kestrel

[abko-auto]

When looking at perf issues came across ObjectDisposed exceptions from Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection!RequestClose

And these ObjectDisposed exception are contributing ~10% of total exceptions. When looked at code KestrelConnection::RequestClose() There is comment about race condition - but this race condition is happening more than 10% of managed exceptions, can we somehow avoid these exceptions?

Entire stack

[bartonjs]

"10% of your managed exceptions" doesn't mean "happens 10% of the time". Presumably a lot of things result in no exceptions whatsoever?

[abko-auto]

Even if I collect perf trace its happening more than 10% of managed exception - doesnt mean 10% of the time but for us that 10% of exceptions in 30 seconds sampling was more than 200 exceptions.

[abko-auto]

Can we have more information about these ObjectDisposed exceptions, I see them all our services? is this something can we fix or we need to live with this? This is one of top 5 exception our application sees, and we would like to get rid of them.

[davidfowl]

Looks like you're calling RequestClose after the connection ends? It's being called from a timer thread and is likely unsynchronized with the connection ending.

[abko-auto]

Thanks @davidfowl for your quick responses and possible solutions.

[abko-auto]

@davidfowl - Isnt 2nd option suggested will have same problem - middleware is already finished and we are about to run registered delegate?

[davidfowl]

@davidfowl - Isnt 2nd option suggested will have same problem - middleware is already finished and we are about to run registered delegate?

No the token will be disposed (see the using var syntax) before the middleware unwinds. As a result, it won't trigger the callback anymore, so the connection abort attempt won't happen.

[abko-auto]

Got it! Was reading the code as plain text. Considering its running as middleware - totally makes sense!
Is this the pattern people use to close connections periodically? What we are trying to do is after every 5 minutes, sending RequestClose() back to the client to avoid sticky connections and balance our traffic evenly.

[davidfowl]

@abko-auto I don't know if people use this pattern. I haven't seen anyone a majority of people ask for it. We have a keep alive timeout by default that can take care of this if you are worried about idle connections. Also if you do have long lived connections, doing it at this level makes it hard to know which connections to timeout because there are only short lived requests on the connection vs a long lived streaming style connection (websocket, server sent events etc).

[abko-auto]

Exactly, after reading existing code and solution you mentioned above realized we are not doing whats expected. We do have services where we expect long lived connection, do we have better solution to close long lived connections after certain time gracefully?

[davidfowl]

I think this pattern is still fine, the only other thing you'd want to add here potentially is the ability for a request to cancel the timeout. That way, code further up the stack that is results in a long running request/connection can cancel the timeout.