Should dotnet have a DNS cache?

[ezsilmar]

One of our applications makes a significant amount of http calls (~1000 per second) to a single external domain using HttpClient. This leads to a high number of getaddrinfo calls (~70 per second) which in turn results in the same number of DNS queries being sent to the infrastructure. Given we have hundreds of instances of this app, it creates significant load on the DNS servers and accounts for more than half of all DNS requests in the datacenter. One important detail is that this DNS record contains ~50 IPs which may make getaddrinfo slower than usual. We made a workaround by providing ConnectCallback and implementing a simple DNS cache which also happen to reduce p99 response time.

What bothers me is that Dns.GetHostAddresses explicitly relies on the assumption that getaddrinfo is fast when called on the same domain name consecutively (see this comment). Basically, it means that dotnet requires OS-level DNS cache when working with high request rates. However, DNS cache is not something provided by default especially on Linux distributions: it is possible to set up but not readily available.

So I wonder what is the official guideline regarding DNS caching, and are there plans to implement DNS cache on the runtime level? Iirc it was provided by now deprecated ServicePoint class in netfw. Thanks!

[MihaZupan]

cc: @antonfirsov @dotnet/ncl

[karelz]

For context, there are scenarios which depend on DNS-resolution load balancing, where we have to ask the OS for DNS resolution on each request.
That said, we have Custom DNS in our plans and extending it with cache would make sense. Using ConnectCallback is a good interm workaround.

Out of curiosity: How did ServicePoint implement DNS cache? Didn't it just rely on OS resolution as well?

[ezsilmar]

I didn't look at the code for this one but there were parameters in ServicePointManager that looked like DNS cache, see this doc. So I believe it used syscalls to make the DNS resolution but it cached the answers, comparing to the current solution that also uses the syscall but doesn't cache.

[antonfirsov]

There is a DNS caching mechnism in ServicePoint indeed:

https://referencesource.microsoft.com/#System/net/System/Net/ServicePoint.cs,1637

The problem is that it times out the entries after DnsRefreshTimeout (default =120 sec), which is incorrect. A proper DNS cache would need to respect the TTL of the DNS records. Unfortunately, the getaddrinfo OS API-s do not return that information, meaning that we would need to implement a DNS resolver in managed code or use a 3rd party library, which is essentially #19443 territory as Karel mentioned.

[MihaZupan]

One of our applications makes a significant amount of http calls (~1000 per second) to a single external domain using HttpClient. This leads to a high number of getaddrinfo calls (~70 per second)

Leaving aside DNS caching, I'm curious why you are creating so many new connections under a steady load?

[ezsilmar]

I re-captured the events and I think the large number of open connections is the consequence of the DNS infra being saturated. Let me try to explain it with the exact numbers from the trace, using Private.InternalDiagnostics.System.Net.Http and System.Net.NameResolution providers.

In the normal situation (DNS infra is not saturated) I get 761 req/s on this specific domain, 6.7 new connections per second, 6.7 dns req/s, and 17 requests queued per second i.e. when there's no available connection in the HttpConnectionPool. The number of opened connections is between 60 and 80. There are two reasons why the connections are re-created: either the request is cancelled on the app side as we have latency constraints (5.5 per second) or the other side breaks ("The response ended prematurely", 0.7 req/s). Notice also that requests are queued more often than the new connections are opened: I think some of them get served by the freed connections while the new ones are opening. Also, "no available connection" situation happens in spikes: there could be up to 15 requests waiting but after the new ones are created the pool is not saturated for the next second or so.

However when DNS infra is saturated (so getaddrinfo becomes slower) I can see the numbers I originally posted. The rate of connections being cancelled stays the same while opening new connections takes longer, so we try to open more new connections, which leads to more load on the DNS making getaddrinfo calls even slower.

[wfurt]

pretty much every distro with systems now has dns cache AFAIK. It seems strange that DNS to local server would be hindering connections to external sites. It feels like this is if worth of investigating.
As @karelz mentioned there are some plans for more functionality improvements but .NET 9 is still year faraway.