A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught)

[Nico-VanHaaster]

Describe the bug

We are using Entity Framework 7 with the Microsoft.Data.SqlClient package version 5.1.1. The issue we are facing is the following stack trace appears in groups in our logs every few hours from all our services that use a common routine for loading a database connection. From the logs these issues are typically cleared up within 30seconds, but during that time there is significant errors (10-15).

These errors are all happening from different services (hosted in a AKS cluster) within the same time framework.

I will add that all the services connect to the Azure SQL Database using Private endpoints, and will stop happening after 15-30 seconds. The interval between the errors is random, anywhere from 30minutes to 4 hours so I am at a complete loss where to look.

Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught)
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Read(Span`1 buffer)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.Data.SqlClient.SNI.SNITCPHandle.Receive(SNIPacket& packet, Int32 timeoutInMilliseconds)
   at ....
ClientConnectionId:4d28a01d-2de1-4540-b1ca-9e9ef9cd9501

Further technical details

Microsoft.Data.SqlClient version: 5.1.1
.NET target: dotnet core 7
SQL Server version: Azure Sql
Operating system: Debian Linux - Docker

[Kaur-Parminder]

@Nico-VanHaaster Thanks for reporting. We will try to reproduce it on our end and update the issue.

[dmytro-gokun]

Looks like we experience the exact same problem in our Azure Container App. The exact same code does not produce this exception when run in Service Fabric environment

[dmytro-gokun]

@Kaur-Parminder Did you have a chance to investigate this?

[DavoudEshtehari]

@dmytro-gokun, It's a different environment configuration and I'd suggest connecting Azure Support for further assistant with it.

[dmytro-gokun]

@DavoudEshtehari After some further investigation, here's my findings:

1. This is a transient network exception;
2. The real difference is not between Service Fabric/Kubernetes, but about OS. When running on ServiceFabric, we use Windows OS and this SqlException than comes back with an SqlError where Number = 10053. When run on Kubernetes/Linux, we get an SqlException where Number = 0.

And this is where the difference happen. When we have 10053, we know it's a transient exception and retry it. For example, EF has this code to find if an exception is transient: https://github.com/dotnet/efcore/blob/release/8.0/src/EFCore.SqlServer/Storage/Internal/SqlServerTransientExceptionDetector.cs

But when we get an exception with Number=0 we do not detect it as transient and have to abort the current operation and write error information to the our log.

So, to me this looks like a SqlClient/Linux bug where SqlException.Number is not populated properly. Do you agree?

@Nico-VanHaaster Do you think this also describes your situation?

[ErikEJ]

Looks related to #1773 and #1902

[dmytro-gokun]

@ErikEJ You are right. And both of those issues are closed. So, the logical question comes: how do we detect transient errors in an OS-independent manner? Is it even possible with the current codebase?

[David-Engel]

In talking with the ODBC driver team, for their Linux implementation, they actually have translations from Linux into the equivalent Windows error codes to try and maintain prior Windows-only behavior under error conditions. So it sounds like this is an enhancement we need to make in our managed SNI layer that's used by non-Windows clients. I bet this would be helped by implementing #649...

[egorshulga]

Let me share my piece of investigation. Recently we observed a spike of errors, which are exactly the same as the topic starter had, and this also impacted our production systems, so we had to take the closest look as we could.

So, we followed the guess, that on linux the code of the underlying problem is not correctly initialized (it is left as 0), thus the built-in retry mechanism simply misses the case when it should be retried. To track it down we added additional logging to AppInsights (which was injected into the ShouldRetryOn method of an ExecutionStrategy of DbContext).

And these are the results: on the image you can see that the SqlException.Errors[0].Number [02] is 0, and that is exactly the property that is used in the built-in retry mechanism of the SqlClient. But we can also see that SqlException.InnerException.InnerException.SocketErrorCode is 10054 [14], and this code is exactly one of the list of retryable codes.

So, it seems that retry mechanism design assumes the SocketErrorCode is copied into SqlError.Number, which obviously does not happen in our case in our environment. May I ask for an advice from the maintainers, what could be the next place to search for the source of the trouble? Where the creation of SqlException is performed?