Periodic Query Spikes and Cache Bypass in dnscrypt-proxy with DoH Server

[vlasiuk-everstake]

Hello,
We are currently using dnscrypt-proxy version 2.1.7 with a custom DoH (DNS-over-HTTPS) server powered by CoreDNS . The configuration file in txt ([dnscrypt-proxy.txt]) dnscrypt-proxy.txt is attached for reference.

Every 12 hours, we observe a significant spike in the number of queries being sent to our DoH server. A screenshot of the query pattern is also attached for clarity.

We increased max_clinets from 250 to 10k but we have an issue [WARNING] Too many incoming connections (max=10000)
Dnscypt-proxy is trying to resolve DNS (doh.****.com) of our DOH server.
Despite enabling caching in dnscrypt-proxy, the queries seem to bypass the cache entirely.

We suspect there might be an issue with the configuration or behavior of dnscrypt-proxy, but we are unable to pinpoint the root cause. Could you help us identify and resolve this issue?

./dnscrypt-proxy -version
2.1.7

./dnscrypt-proxy -check
[2025-02-21 12:51:17] [NOTICE] dnscrypt-proxy 2.1.7
[2025-02-21 12:51:17] [NOTICE] Source [coredns-resolvers] loaded
[2025-02-21 12:51:17] [NOTICE] Configuration successfully checked

./dnscrypt-proxy -resolve goole.com
Resolving [google.com] using 127.0.0.1 port 53


Resolver : 172.217.40.22

Canonical name: google.com.

IPv4 addresses: 142.251.39.110
IPv6 addresses: 2a00:1450:400e:80f::200e

Name servers : ns3.google.com., ns4.google.com., ns2.google.com., ns1.google.com.
DNSSEC signed : no
Mail servers : 1 mail servers found

HTTPS alias : -
HTTPS info : [alpn]=[h2,h3]

Host info : -
TXT records : google-site-verification=4ibFUgB-wXLQ_S7vsXVomSTVamuOXBiVAzpR5IZ87D0, apple-domain-verification=30afIBcvSuDV2PLX, facebook-domain-verification=22rm551cu4k0ab0bxsw536tlds4h95, google-site-verification=TV9-DBe4R80X4v0M4U_bd_J9cpOJM0nikft0jAgjmsQ, docusign=05958488-4752-4ef2-95eb-aa7ba8a3bd0e, v=spf1 include:_spf.google.com ~all, cisco-ci-domain-verification=479146de172eb01ddee38b1a455ab9e8bb51542ddd7f1fa298557dfa7b22d963, docusign=1b0a6754-49b1-4db5-8540-d2c12664b289, google-site-verification=wD8N7i1JTNTkezJ49swvWW48f8_9xveREV4oB-0Hf5o, onetrust-domain-verification=de01ed21f2fa4d8781cbc3ffb89cf4ef, MS=E4A68B9AB2BB9670BCE15412F62916164C0B20BB, globalsign-smime-dv=CDYX+XFHUw2wml6/Gb8+59BsH31KzUr6c1l2BPvqKX8=

[jedisct1]

(We?)

Looks like even Cloudflare DNS is having trouble resolving the name of that DoH server.

Check the configuration of the authoritative DNS server for that domain, and that the bootstrap servers are not firewalled.

Or, in order to not have to resolve the name, include the IP addresses in the stamp or use DNSCrypt.

[sergey2000k]

Same config, except, of course, different DoH server.
No clients connected to the server, so No DNS queries from clients at all.
No DNS queries from the server itself.
Bootstrap servers (9.9.9.9, 8.8.8.8, 1.1.1.1) are not firewalled.

Same problem:
Every 12 hours since the moment dnscrypt-proxy started (except the moment right after starting) there are lots of repeated messages in the log:

[2025-03-25 10:11:22] [WARNING] Too many incoming connections (max=10000)
[2025-03-25 10:11:22] [INFO] Unable to resolve [doh***] using resolver [127.0.0.153:53] (tcp): EOF
[2025-03-25 10:11:22] [NOTICE] Resolving server host [doh***] using bootstrap resolvers over udp
[2025-03-25 10:11:22] [INFO] Unable to resolve [doh***] using resolver [127.0.0.153:53] (udp): read udp 127.0.0.1:41095->127.0.0.153:53: i/o timeout

But, every 4 hours (except every third 4-hour, i.e. except the error every 12 hours) since the moment dnscrypt-proxy started there are brief successful messages like:

[2025-03-25 02:11:14] [NOTICE] queries for [doh***]
[2025-03-25 02:11:15] [INFO] [doh***] TLS version: 304 - Protocol: h2 - Cipher suite: 4865
[2025-03-25 02:11:15] [INFO] [doh***] OK - rtt: 49ms
...
[2025-03-25 06:11:15] [NOTICE] queries for [doh***]
[2025-03-25 06:11:20] [INFO] [doh***] TLS version: 304 - Protocol: h2 - Cipher suite: 4865
[2025-03-25 06:11:20] [INFO] [doh***] OK - rtt: 91ms

[lifenjoiner]

Periodic Query Spikes

every 4 hours successful messages

It is

dnscrypt-proxy/dnscrypt-proxy/example-dnscrypt-proxy.toml

Lines 193 to 195 in 5d2519e

	## Delay, in minutes, after which certificates are reloaded
	cert_refresh_delay = 240

, for

dnscrypt-proxy/dnscrypt-proxy/example-dnscrypt-proxy.toml

Lines 150 to 160 in 5d2519e

	## Load-balancing strategy: 'p2' (default), 'ph', 'p<n>', 'first' or 'random'
	## Randomly choose 1 of the fastest 2, half, n, 1 or all live servers by latency.
	## The response quality still depends on the server itself.
	# lb_strategy = 'p2'
	## Set to `true` to constantly try to estimate the latency of all the resolvers
	## and adjust the load-balancing parameters accordingly, or to `false` to disable.
	## Default is `true` that makes 'p2' `lb_strategy` work well.
	# lb_estimator = true

, by restarting server rankings.

Cache Bypass

The above is not cache bypass. It is the connection delays for dnscrypt-proxy to servers.

A DoH sdns stamp without IP shipped requires host name to be resolved. And it uses a standalone cache which is not controled by the config.
Ref: https://dnscrypt.info/stamps/

Valid cache items for clients should follow:

dnscrypt-proxy/dnscrypt-proxy/example-dnscrypt-proxy.toml

Lines 408 to 439 in 5d2519e

	###########################
	# DNS cache #
	###########################
	## Enable a DNS cache to reduce latency and outgoing traffic
	cache = true
	## Cache size
	cache_size = 4096
	## Minimum TTL for cached entries
	cache_min_ttl = 2400
	## Maximum TTL for cached entries
	cache_max_ttl = 86400
	## Minimum TTL for negatively cached entries
	cache_neg_min_ttl = 60
	## Maximum TTL for negatively cached entries
	cache_neg_max_ttl = 600

Every 12 hours

It is a special value. Checked the source code. Still no clue what happened.

No clients connected to the server

Good start.
Let's use the single variable method and see if we can find the problem.
Can you try the default config with just an idle port and dnscrypt_servers = false?

dnscrypt-proxy/dnscrypt-proxy/example-dnscrypt-proxy.toml

Line 42 in 5d2519e

listen_addresses = ['127.0.0.1:53']

dnscrypt-proxy/dnscrypt-proxy/example-dnscrypt-proxy.toml

Line 67 in 5d2519e

dnscrypt_servers = true

Your OS and its settings?

[jedisct1]

What you just did is restore the behavior of old versions where DNS queries to resolve DoH names were not encrypted.
Some people complained about that. So, queries are now first sent to the proxy itself, assuming that when a DoH server that doesn't specify IP addresses is used, there is also at least a DNSCrypt server or a DoH server with IPs also configured.

[jedisct1]

The code in git may improve the scenario where only one server is used and it's a DoH one.

Something we may also want to do is add some jitter to the refresh time, so that all resolvers are not refreshed around the same time.

[sergey2000k]

Thank you for the explanation!

What you just did is restore the behavior of old versions where DNS queries to resolve DoH names were not encrypted.

Yes, I understand that my "solution" isn't a secured one.

Something we may also want to do is add some jitter to the refresh time

If you know, could you please explain why there is a problem "Too many incoming connections" along with "Unable to resolve..." happen exactly every 12 hours, and there is no problem if just do, for example, the command 'nslookup doh***' for a lot of times (without cache).

PS. I use only one DoH server and no DNSCrypt

[jedisct1]

After the IP address of a resolver expires, it has to be resolved again so that the server can keep being used. If there's only one server configured, it can only ask itself, but since it doesn't work because its IP expired...

[lifenjoiner]

Make sense.
People usually emphasize what they think is important and ignore what really counts.
But the failure of fallback servers is weird, and is more likely due to the host name itself ... @vlasiuk-everstake
The problem is out of service, no cache bypass, and periodic query spike is just the way it shows up.
@jedisct1 has added improvements for this only one DoH server special case.

BTW, a router does not need to use a DNS server run by itself. So that, the system DNS will be those from ISP or the upstream router, yielding more options, and making it more robust.
Notes: Only for source and DoH host names. That's the bootstrap. Users' queries always go to the configured encrypted servers.