Performance Issues with SpoofCNAMEAction under High Load in dnsdist 2.0.0-alpha1

[Zqsdoo]

Hello,

I've been experimenting with dnsdist 2.0.0-alpha1, trying to use SpoofCNAMEAction under high traffic conditions, but it's not working as expected.
Specifically, with around 5,000 rules like the one below:

addAction("domain", SpoofCNAMEAction("target"))

I observe a significant performance drop.

Without any rules, my server can handle up to 1 million requests per second. However, with the spoofing rules in place, the throughput drops to around 100,000 requests per second.

My question is: What would be the best approach to efficiently spoof CNAME responses under heavy load in dnsdist?

Do you have any suggestions or best practices for handling this kind of scenario?

Thanks in advance!

[rgacogne]

It's impossible to be sure without seeing your actual full configuration, but I expect the performance impact is coming from the high number of rules you are defining. The way it works is that rules are evaluated one by one, so doing 5000 DNS suffixes comparison is going to hurt. If you are using the same target for all domains you can replace these 5000 by a single SuffixMatchNodeRule, like this:

mySMN = newSuffixMatchNode()
-- add a single name, you would need to do this for all the domains you want to match
mySMN:add(newDNSName("powerdns.com."))

addAction(SuffixMatchNodeRule(mySMN), SpoofCNAMEAction("target"))

If you need to have different targets based on the domain that was matched, I would advise looking at the Key-Value stores supported by dnsdist (CDB and LMDB): https://www.dnsdist.org/reference/kvs.html This would also make it a lot easier to handle updates to your list, since KVS can be automatically reloaded.

[Zqsdoo]

Would you be able to downgrade to rc2 to see if the latency goes back to the way it was before? If you need -rc2 packages you can find them there: https://downloads.powerdns.com/autobuilt_browser/#/dnsdist/2.0.0-rc2.0.reldnsdist20x.gb589bc8ee

I tried it, but it didn't seem to change anything. Maybe it's an issue on my side. I'm still investigating

[rgacogne]

Thanks a lot for testing, this is very much appreciated! One thing is that the latency reported by DNSdist takes a bunch of queries to stabilize after startup, so it might be normal for the latency to slowly increase just after a restart. It really depends on the number of QPS you are processing: with a lot of them the latency stabilizes very quickly.

[Zqsdoo]

Thanks a lot for testing, this is very much appreciated! One thing is that the latency reported by DNSdist takes a bunch of queries to stabilize after startup, so it might be normal for the latency to slowly increase just after a restart. It really depends on the number of QPS you are processing: with a lot of them the latency stabilizes very quickly.

To show you i took ~1w before and after
Upon further checking, we can clearly see that we were on that version before the upgrade :

And it changed afterward :

Just to add, I tried both 2.0.0-rc2 and 2.0.0-rc1, and the issue is the same, i really think it might be caused by something in our infrastructure, but it's strange because it appeared exactly at the moment of the upgrade.
Honestly, that’s quite odd

[rgacogne]

If I'm not mistaken your dnsdist setup is in front of recursive resolvers, right? If that's the case the variation might be due to a lot of things, I have seen cases where a small change in network routing toward one big actor (Cloudflare, Route 53, Akamai, etc..) causes such a small variation. Have you looked at the metrics reported by the backend themselves?

[Zqsdoo]

n. Have you looked at the metrics reported by the backend themselves?

Thank you for your response. I have already checked them, but maybe not in an overall graph. It’s true that there is a big difference in the average latency, but it looks like the difference appeared when I upgraded the dnsdist instance
According to the following graph, my downgrade from yesterday clearly affected this part. I’ll try upgrading them again and will give you feedback

After some investigation and upgrading all instances to the latest version, it looks like things are returning to a normal state

[Zqsdoo]

Thanks for the fast response
My actual configuration look's like that :

-- Set the local address for the service
setLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

-- Add the local for DNS-over-HTTP/3 do multiple to be better
addDOH3Local('IPV6', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/.key',{internalPipeBufferSize=1048576,reusePort=true})

addDOH3Local('IPV4', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key',{internalPipeBufferSize=1048576,reusePort=true})


-- Add the local for DNS-over-HTTP/<2 (dont forget to add the header to specify that we have DNS-over-HTTP/3 enabled)
addDOHLocal('IPV6','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', "/dns", {customResponseHeaders={["alt-svc"]='h3=\":443\"'},internalPipeBufferSize=1048576,reusePort=true}) -- Internal pipe buffer value is the recommanded for high load

addDOHLocal('IPV4','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', "/dns", {customResponseHeaders={["alt-svc"]='h3=\":443\"'},internalPipeBufferSize=1048576,reusePort=true}) -- Internal pipe buffer value is the recommanded for high load

-- Add the local for DNS-over-QUIC (DoQ) (adding congestion control algo with brr for this on)
addDOQLocal('IPV6', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', {internalPipeBufferSize=1048576,congestionControlAlgo="bbr",reusePort=true})
addDOQLocal('IPV4', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', {internalPipeBufferSize=1048576,congestionControlAlgo="bbr",reusePort=true})

-- Maybe add TLSlocal
addTLSLocal('IPV6','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key',{reusePort=true})
addTLSLocal('IPV4','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key',{reusePort=true})




-- Set the Access Control List (ACL) to allow all IPv4 and IPv6 addresses
setACL({"0.0.0.0/0", "::/0"})

-- Set the DNS Resolver (if you have only one backend you can use the same multiple time to run it on multiple thread)
newServer({address="IPV4 backend", name="example-back"})
newServer({address="IPV6 backend", name="example-back"})

newServer({address="IPV4 backend", name="example-back1"})
newServer({address="IPV6 backend", name="example-back1"})

newServer({address="IPV4 backend", name="example-back2"})
newServer({address="IPV6 backend", name="example-back2"})

newServer({address="IPV4 backend", name="example-back3"})
newServer({address="IPV6 backend", name="example-back3."})

newServer({address="IPV4 backend", name="example-back4"})
newServer({address="IPV6 backend", name="example-back4t"})

newServer({address="IPV4 backend", name="example-back5"})
newServer({address="IPV6 backend", name="example-back5"})

newServer({address="IPV4 backend", name="example-back6"})
newServer({address="IPV6 backend", name="example-back6."})

newServer({address="IPV4 backend", name="example-back7"})
newServer({address="IPV6 backend", name="example-back7"})

newServer({address="IPV4 backend", name="example-back8"})
newServer({address="IPV6 backend", name="example-back8"})

newServer({address="IPV4 backend", name="example-back9"})
newServer({address="IPV6 backend", name="example-back9"})

newServer({address="IPV4 backend", name="example-back10"})
newServer({address="IPV6 backend", name="example-back10"})

newServer({address="IPV4 backend", name="example-back11"})
newServer({address="IPV6 backend", name="example-back11"})

newServer({address="IPV4 backend", name="example-back12"})
newServer({address="IPV6 backend", name="example-back12"})

newServer({address="IPV4 backend", name="example-back13"})
newServer({address="IPV6 backend", name="example-back13"})


webserver("Your local ipv4 addr:8083")
setWebserverConfig({
  password="password",
  apiKey="apikey)",
  acl="ipv4"
})

-- UDP Configuration
setMaxUDPOutstanding(65535)
setRingBuffersSize(65535)
-- Set this to 0 0 to use the systemctl configuration
setUDPSocketBufferSizes(0, 0)
-- Increase thread count for TCP client connections (10 is the default value ) 
setMaxTCPClientThreads(10)
-- https://www.dnsdist.org/advanced/tuning.html#outgoing-doh
setOutgoingDoHWorkerThreads(8)
-- https://www.dnsdist.org/reference/tuning.html#setMaxTCPQueuedConnections
setMaxTCPQueuedConnections(0) --0 mean unlimited
setTCPInternalPipeBufferSize(1048576)


-- Set the cache size (20000000 entries) and the cache policy (https://www.dnsdist.org/guides/cache.html) for 16GB ram
pc = newPacketCache(20000000, {maxTTL=86400, minTTL=0, temporaryFailureTTL=60, staleTTL=60, dontAge=false})
getPool(""):setCache(pc)

-- BPF Configuration (Packet Filtering)
-- Define a new BPF filter with specific limits for IPv4, IPv6, and query names
bpf = newBPFFilter({ipv4MaxItems=500000, ipv6MaxItems=500000, qnamesMaxItems=500000})
setDefaultBPFFilter(bpf)
-- Create a dynamic block rules group (used for rate limiting and blocking)
local dbr = dynBlockRulesGroup()

function maintenance()
  dbr:apply()
end

-- Set the control socket
controlSocket("127.0.0.1:5199")
setKey("To set using makeKey()")

I precise that i run under 6.8.0 kernel with some tuning with sysctl value
And to be honest, I mainly have a lot of different domains that redirect to the same CNAME. Based on that, I'll explore your solution.
Don't hesitate to help me with the configuration to optimize it 😄

[rgacogne]

At a quick glance your configuration looks good. You might be able to optimize the dnsdist <-> backend path, especially if your backends support TCP out-of-order processing: https://www.dnsdist.org/advanced/out-of-order.html
Please let us know if my suggestion worked as expected, as we might want to investigate what is happening if it does not, and otherwise it can be useful to others users :)

[Zqsdoo]

I actually use pdns_recursor as the backend, and based on the performance tuning guide, I believe that I could enable out-of-order processing. I'll test that later, but since my main objective is to optimize the UDP part and DOH, these other features will be a bonus. However, they might slow things down, so I’ll need to proceed with caution.

Here’s my current setup:

dnssec:
  trustanchorfile: /usr/share/dns/root.key 
recursor: 
  hint_file: /usr/share/dns/root.hints
  include_dir: /etc/powerdns/recursor.d

  # Tuning
  threads: 8              # More threads for query handling (scales with CPU cores)
  tcp_threads: 8           # Keep this lower unless many TCP queries are expected
  max_mthreads: 32768      # Increase for high concurrency, allow more lightweight threads
  stack_cache_size: 65536  # Larger for deeper stacks if needed

incoming:
  pdns_distributes_queries: true
  distributor_threads: 8
  reuseport: true
  max_tcp_clients: 16384   # High TCP load tolerance

  max_concurrent_requests_per_tcp_connection: 15000

outgoing:
  # source_address: 0.0.0.0 (default)

recordcache: 
  max_entries: 10000000    # Boosted for high traffic (8.5GB RAM est.)
  max_ttl: 86400           # Cache records longer

packetcache:
  max_entries: 4000000     # 500 bytes each, ~2 GB memory

webservice:
  address: ipv4
  port: port
  api_key: 'apikey'
  password: 'password'
  allow_from: ['ipv4']
  webserver: true

I’m also planning to add the maxInFlight parameter to both the backend and the addlocal section. I’ll need to think about a reasonable value for those settings. Once I have that figured out, I'll send an update here. 😊

[rishu007]

@rgacogne can we dynamically return CNAME from dnsdsit?
for example "-
Query:-
test--hostname.domain.com
expected cname
test.hostname
is this possible like get part of query and generate dynamic cname ?