DNS-privacy with PowerDNS

[sveeke]

First of all, Hi!

Nothing to hide is a privacy infrastructure provider in the Netherlands (https://nothingtohide.nl/) and we're working on making DNS more resistant to timeless timing attacks and correlation attacks. This is a important feature for anonymity networks, especially with the current state of surveillance in many countries.

This entails much more than just adding functionality to dnsdist/Recursor, but by far the biggest challenges are how to add 3 features to dnsdist/Recursor. These are explained below and I hope some people with much more knowledgable than me can help with this :).

It's much appreciated!

1 Random TTL to clients (dnsdist)

DNS frontend (dnsdist) answers client requests with a random TTL within a configured range.

• dnsdist has a built-in action "LimitTTLResponseAction(min[, max[, types]])" [1] which allows for a fixed TTL. However, this is not random.
• I have tried scripting this, and an acquaintance also used a paid AI model on it, but in all cases, it resulted in various errors. So first, the following question:
• Can this random TTL be added to dnsdist using Lua or FFI? What is the best approach?

[1]

local fixedTTL = 123
addResponseAction(AllRule(), LimitTTLResponseAction(fixedTTL, fixedTTL))

2 DNS hot cache (Recursor)

The DNS cache refreshes all cached records before they expire.

• In a conversation with Habbie at NLNOG, I mentioned that I'd like to use dnsdist for this, but Habbie rightly pointed out that dnsdist doesn't have "hot refresh" functionality and isn't "aware" enough for this (meaning it can't be added with Lua).
• Recursor does have a "hot refresh" functionality [2], but it only refreshes when a cached record reaches a certain percentage of its original TTL and is triggered by a new request for that specific record.
• Is it possible to have Recursor refresh all records regardless of whether they are requested? For example, "if TTL < 20% then refresh record." This is essentially the most essential feature, and I have no idea how this could work.
• In theory, I think you could include "refresh-on-ttl-perc = 50" in the configuration and run a tool that tries to resolve lists of domains. But this is extremely messy: 1) it works less well for shorter TTLs, 2) it greatly increases the number of outgoing queries, and 3) it doesn't retain records that aren't on your preload list (and that's precisely one of the primary countermeasures against timeless timing attacks in the context of Tor anonymity).

[2] https://blog.powerdns.com/2022/04/28/refreshing-of-almost-expired-records-keeping-the-cache-hot

3 Random TTL in cache (Recursor)

The DNS cache contains random TTL values, depending on a record's original TTL value.

if ORIGINAL_TTL < 300 seconds then TTL = RANDOM(600-900 seconds) 
if ORIGINAL_TTL > 300 seconds then TTL = RANDOM(3000-4000 seconds)

• Since dnsdist doesn't have a hot cache, I'm assuming Recursor's cache for this.
• I think this is possible with Lua using math.random and the postresolve hook (because this happens directly after resolution, but before adding to the cache). I've tried scripting this, but although pdns_recursor works and runs fine (after 100 tries to get this right), this mechanism doesn't seem to work at all. Any idea what I'm doing wrong?
• Are there better, faster, or more efficient ways to do this?

/usr/local/etc/pdns/recursor.yml:

incoming:
  allow_from:
  - 0.0.0.0/0
  listen:
  - 145.220.10.129:5300
  reuseport: true

recordcache:
  max_entries: 10000000

logging:
  loglevel: 9

recursor:
  lua_config_file: /usr/local/etc/pdns/recursor.lua

/usr/local/etc/pdns/recursor.lua:

-- the os.time addition was recommended by a snippet in some git repo, but may not be as useful in this case (need to test impact on performance).
math.randomseed(os.time()) 

function randomizeTTL(originalTTL)
    if originalTTL <= 300 then
        return math.random(600, 900)
    else
        return math.random(3000, 4000)
    end
end

function postresolve(remoteip, domain, qtype, records, origrcode)
    if not records then
        return origrcode, records
    end

    for i, record in ipairs(records) do
        if record.ttl then
            local originalTTL = record.ttl
            local newTTL = randomizeTTL(originalTTL)
            record.ttl = newTTL
        end
    end
    return origrcode, records
end

dig just consistently returns the cached original TTLs, so my fiddling with Lua isn't working. And to be clear: I'm not a programmer, so with my limited shell scripting experience, I've scoured the internet for examples in git repos and documentation. This is what came out of it after much trial and error.

4 Domain feedback loop (for information)

This isn't so much a PowerDNS software feature, but I thought I'd share it anyway in case people like getting some more background information about how the cache is reloaded.

I'm building a tool to automate a weekly domain feedback loop, in order to sync the preload cache between recursors and the make clearing the cache (rebooting for kernel updates) possible without losing the valuable cache entries. The idea is to grow the preload list with actively used records. Roughly as follows:

1. Once a week, PowerDNS Recursor's "rec_control dump-cache" will be used to write a copy of the cache to a file.
2. All superfluous data will be stripped from that file, leaving only the domain names for A, AAAA, CNAME, MX, and TXT records (I don't think NS is very useful to include here).
3. There's a 'master' preload list to which new domain names will be added if they don't already exist.
4. Standard "X million domain lists" will also be downloaded to add those domain names, if they don't already exist.
5. Finally, I still need to devise a mechanism that checks if domain names are resolvable, and if not (after, say, 2 checks with some time in between), removes them from the list. Otherwise, over time, tens of thousands of 'ghost domains' will accumulate.

[omoerbeek]

I will look at the other point later, but to answer one of your questions:

For randomizaition of TTLs in Lua, the following script does "work", but on the other hand is extremely broken, as all records in the same recordset (same name and type) should have the same TTL. Also I expect increasing TTLs compared to the original can cause bad side effects. SO DON'T USE IN PRODUCTION!.

function randomizeTTL(originalTTL)
    if originalTTL <= 300 then
        return math.random(600, 900)
    else
        return math.random(3000, 4000)
    end
end

function postresolve(dq)
    records = dq:getRecords()
    for i, record in pairs(records) do
            local newTTL = randomizeTTL(record.ttl)
            record.ttl = newTTL
    end
    dq:setRecords(records)
    return true
end

[sveeke]

Thanks for the help! But sadly I can't make it work. It loads just fine, but doesn't seem to do anything.

1. I added print("Lua file loaded succesfully") in recursor.lua to make sure recursor.lua actually is loaded. When I start pdns_recursor --daemon=no --loglevel=9 --config-dir=/usr/local/etc/pdns it shows the printed text, indicating that loading the lua file actually works.
2. I added some more prints to your suggested snippet to check whether the functions get executed:

function randomizeTTL(originalTTL)
    if originalTTL <= 300 then
        print("########## lower than 300")
        return math.random(600, 900)
    else
        print("########## higher than 300")
        return math.random(3000, 4000)
    end
end

function postresolve(dq)
    records = dq:getRecords()
    for i, record in pairs(records) do
            local newTTL = randomizeTTL(record.ttl)
            record.ttl = newTTL
            print("########## modified TTL")
    end
    dq:setRecords(records)
    return true
end

But none of the prints in these functions get printed, which makes me think it's not doing anything. As far as I understand, I don't need to call function postresolve(dq) manually, because postresolve is a hook function that Recursor calls automatically (but correct me if I am wrong).

Am I missing something?

edit:
For completeness sake: I use PowerDNS Recursor 5.2.0 on FreeBSD 14.2 and this version is compiled with Lua support.

# pdns_recursor --version
PowerDNS Recursor 5.2.0 (C) PowerDNS.COM BV
Using 64-bits mode. Built using clang 18.1.6 (https://github.com/llvm/llvm-project.git llvmorg-18.1.6-0-g1118c2e05e67) on Jun  3 2025 15:48:34 by root@142amd64-quarterly-job-21.
PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Features: libcrypto-ecdsa libcrypto-ed25519 libcrypto-ed448 libcrypto-eddsa lua nod protobuf curl scrypt
Configured with: " '--sysconfdir=/usr/local/etc/pdns' '--with-libsodium=no' '--with-service-user=pdns' '--with-service-group=pdns_recursor' '--without-net-snmp' '--without-systemd' '--disable-dnstap' '--prefix=/usr/local' '--localstatedir=/var' '--mandir=/usr/local/share/man' '--disable-silent-rules' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd14.2' 'build_alias=amd64-portbld-freebsd14.2' 'CC=cc' 'CFLAGS=-O2 -pipe  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LDFLAGS=  -fstack-protector-strong -L/usr/local/lib ' 'LIBS=' 'CPPFLAGS=-isystem /usr/local/include' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include ' 'PKG_CONFIG=pkgconf' 'PKG_CONFIG_LIBDIR=/wrkdirs/usr/ports/dns/powerdns-recursor/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig' 'PYTHON=/usr/local/bin/python3.11'"

[omoerbeek]

You need to load your script using lua_dns_script, not lua_config_file.

[sveeke]

It works! Thanks for that crucial suggestion.

is extremely broken, as all records in the same recordset (same name and type) should have the same TTL

Yes I notice that when a query returns multiple records, they all have different TTLs. For example:

;; ANSWER SECTION:
powerdns.com.           824     IN      A       199.60.103.61
powerdns.com.           672     IN      A       199.60.103.161

But why would this matter/should they have the same TTL? And would it be possible to randomize a single TTL for all records in a domain instead of doing that on a record basis?

Maybe I'm overthinking the potential solution, but I assume normally not all records for a domain are retrieved at once in all situations. So when a record for a domain is requested, Recursor could (if posisble) just always gather pretty much all record types at once and apply a single TTL value to all of the records for that domain. This sounds relatively low-complexity. Also from the perspective of the primary use-case (and preloading the domains), caching (and refreshing) multiple record types at once is great.

Or another solution might be that for each new records some logic is needed to gather the remaining TTL in the cache for the records belonging to that same domain, before applying that remaining TTL value to the newly requested record. But this sounds fairly complex/cumbersome.

Also I expect increasing TTLs compared to the original can cause bad side effects

For coming up with the concept behind this setup I also spoke with a few researchers specializing in DNS and internet anonymity and that way I already came up with two downsides. But there could be more :).

1. Cached record might get stale during the extended TTL period (e.g. in the case of a failover scenario). This might lead to temporary unreachable websites.
2. Caching records for a longer period of time lowers the effectiveness of utilizing DNS for managing load/load balancing. Especially if a significant part of the Tor network would use these cached records.

It's probably impossible in practice to mitigate these downsides in all cases, and to a certain degree this is a lesser issue than being susceptible to timeless timing attacks and correlation attacks. That being said I also thought of a mechanic to lower this impact of increased TTLs significantly. I just didn't include it in this discussion because I thought it would be better to keep things simple for now, and then when the basics work move on from there.

A large chunk of DNS requests on the Tor network (and in general) are actually about a fairly limited amount of domains by a even smaller amount of companies. So I was thinking of making a list of the domains used by the usual suspects, load the file in to Recursor and add a if statement with a much lower random TTL (for example between 5 and 10 minutes). This way the most popular internet services (that pretty much all use extremely low TTL values for their domains) wouldn't be as impacted by us serving increased TTL values. And similarly users are less impacted by stale records or failover shenanigans.

Do you think something like this would be beneficial/is possible? Based on some snippets I found, it might look something like this (not tested, probably contains bugs):

local specialDomains = {}

function loadDomains()
    local file = io.open("/usr/local/etc/pdns/domains.txt", "r")
    if file then
        for line in file:lines() do
            -- remove whitespace, comments and convert to lowercase
            local domain = line:match("^%s*([^#]-)%s*$"):lower()
            -- if domain is not an emptu string, then add it to the hash table specialDomains
            if domain ~= "" then
                specialDomains[domain] = true
            end
        end
        file:close()
    else
        print("Lua script error: /usr/local/etc/pdns/domains.txt file not found")
    end
end

loadDomains()

function isDomainInList(queryDomain)
    local domain = queryDomain:lower()
    if specialDomains[domain] then
        return true
    else
        return false
    end
end

function randomizeTTL(originalTTL, queryDomain)
    -- Check if domain is in the special list
    if isDomainInList(queryDomain) then
        return math.random(300, 600)
    elseif originalTTL <= 300 then
        return math.random(600, 900)
    else
        return math.random(3000, 4000)
    end
end

function postresolve(dq)
    local queryDomain = dq.qname:toString()
    records = dq:getRecords()
    
    for i, record in pairs(records) do
        local newTTL = randomizeTTL(record.ttl, queryDomain)
        record.ttl = newTTL
    end
    
    dq:setRecords(records)
    return true
end

By the way, thanks a lot for replying and helping! I appreciate it a lot. And I'm also looking forward to your other remarks and thoughts :).

[omoerbeek]

https://datatracker.ietf.org/doc/html/rfc2181#section-5.2 explains why resource records in a resource record set (defined as having the same name and type) must have the same TTL. Clients are even supposed to drop RRSets having divergent TTLs.
So you'll need some logic in your Lua to code to sensure you achieve that. A simple map from name and type to TTL assigned earlier can do the trick.
I would consider to never increase the TTL, but return a random TTL between 1 and the original TTL.

[sveeke]

Thanks, that spec is compelling. There might be enough clients that actually implemented this check to make it a problem.

About lowering TTL:
Lowering the cache TTL is perfectly possible from a anonymity standpoint, but it will increase DNS requests to upstream nameservers considerably. So our considerations when coming up with a TTL of 600-900 and 3000-4000 was to be a 'good DNS citizen' so to speak.

As an example:

• Renewing 5M cached records with an average TTL of 1300 will result in ~4k requests.
• Renewing 5M cached records with an average TTL of 300 will result in ~12k requests.

And of course this would be worse with more cached records and/or a lower TTL. Do you think increasing upstream requests is a problem for the Domain Name system in general? Or are these numbers rounding errors in the grand scheme of things?

[sveeke]

And regarding consistent TTLs: I think it works as it should now! In the sense that every RRSet has its own TTL value.

With my own tinkering I got userdata object/string and some other issues so I had help from an LLM to create this piece of code. Does this look sane?

local rrsetTTLMap = {}

function randomizeTTL(originalTTL)
    if originalTTL <= 300 then
        return math.random(600, 900)
    else
        return math.random(3000, 4000)
    end
end

function postresolve(dq)
    local records = dq:getRecords()
    -- clear the map for this query
    rrsetTTLMap = {}
    
    for i, record in pairs(records) do
        -- create a unique key for this RRSet (name + type)
        local rrsetKey = tostring(record.name) .. "|" .. tostring(record.type)
        -- check if we've already assigned a TTL for this RRSet and if not assign random TTL
        if rrsetTTLMap[rrsetKey] == nil then
            rrsetTTLMap[rrsetKey] = randomizeTTL(record.ttl)
        end
        -- Assign the consistent TTL for this RRSet
        record.ttl = rrsetTTLMap[rrsetKey]
    end
    dq:setRecords(records)
    return true
end

[omoerbeek]

This might work indeed. Note you still might encounter issues if the RRSIG RR TTL does not match the TTL of the RRSet it signs (see https://www.rfc-editor.org/rfc/rfc4034 section 3 last sentence). There might be other edge cases as well.

Some randomn thoughts/remarks:

As for refreshing all the records in the cache, I would argue against that, as it will fill the cache with useless records: records that will never be re-queried by clients (not only pseudo random attacks, but also genuine transient records. There's also the case of records that do not exist anymore (are not served anymore by auths): re-querying those wil fill up the negative cache with what is likely useless data as the original record was meant to be transient.

Add to that that the cache is a finite resource: if it fills up, the recursor still has to evict records.

Now I think a mechanism that makes a difference between the TTLs sent to the client and the one internally used for cache expiration could be effective: the internal TTLs still would be the one received from the auth. The one sent to the client could still be much lower, even 1 for all maybe records (I did not think this through 100%). This would require internal modification to the record cache and/or packet cache, though. Maybe dnsdist would be easier for this approach.

Generally speaking, I believe what you are trying to achieve is hard. The proposed methods might help a bit achieving what you are trying to do, but a big gap remains: to see if a name was in the cache, it is enough to measure the response times. A cache hit will be orders of magnitude faster than a cache miss. If you do not introduce delays in your cache hit respones it still will be trivial to tell them apart from cache hits. But introducing delays will create big queues of work in rec or dnsdist, so that has big drawbacks.

[sveeke]

Thanks for your thoughts!

As for refreshing all the records in the cache, I would argue against that, as it will fill the cache with useless records: records that will never be re-queried by clients

The thing is: caching some x million additional records isn't so bad when it increases cache hit %. There will be some records that will rarely of even never be queried, but that's still better than the alternative when your government actively targets you. You actually give an argument for this later in your reply: "The proposed methods might help a bit achieving what you are trying to do, but a big gap remains: to see if a name was in the cache, it is enough to measure the response times. A cache hit will be orders of magnitude faster than a cache miss."

This is exactly the reason why as much records as possible should be cached. The higher the cache hit rate, the less likely the earlier mentioned attacks can be successfully executed. 100% cache hit rate will be impossible, but it should be possible to get close.

We also wrote about this about a year ago in 2.1 DNS cache misses of this blog: https://nothingtohide.nl/blog/improving-dns-privacy/.

There's also the case of records that do not exist anymore (are not served anymore by auths)

Yes this issue needs some more work. I'm trying to think of some way to remove those from the preload list every week or so. It shouldn't just infinitely grow with millions (when time passes) of non-existing records, but it wouldn't matter much if they were cached for a week or so before they are purged from the preload list (effectively making sure they are not added to the cache on weekly 'reset' anymore).

Originally we thought about a 'purge timer' that has some overlap (but not 100%) with this issue. We wrote this about it last year in the same blog mentioned before:

Every time a domain’s record is requested, the ‘purge timer’ will be reset to zero. If a domain’s record has not been requested for the retention period, it gets purged from the cache to not let it linger around indefinitely. These same mechanics apply to the preload list.

But that also won't be easy to implement I imagine (let alone in Lua?).

Add to that that the cache is a finite resource: if it fills up, the recursor still has to evict records.

It's finite, as everything is finite with regards to software/servers. But is this really a consideration nowadays? If my estimation based on my current cache memory consumption is correct every 100M records take between 50-80 GB of RAM? That (or many more) shouldn't be a problem at all (memory capacity wise at least).

Now I think a mechanism that makes a difference between the TTLs sent to the client and the one internally used for cache expiration could be effective: the internal TTLs still would be the one received from the auth.

This makes attacks easier sadly. It only works when the outgoing queries (for the internally used cache) aren't predictable and the TTLs to clients aren't predictable either. It probably wouldn't matter if the TTLs were more in line with the original TTLs than is the case with the earlier mentioned 600-900/3000-4000 (which to be fair are quite long!). But they shouldn't be the exact TTL.

Also I haven't seen many valid use-cases for using <=60s TTLs (and many arguments against it), while it does make the tracking of people's internet behavior on a massive scale vastly easier and more fine-grained. Respecting insanely low TTLs doesn't make any sense to me if I'm honest. That being said, there is a lot of ground between a TTL of 60 and ~750 so shortening it considerably should be fine.

The one sent to the client could still be much lower, even 1 for all maybe records (I did not think this through 100%)

Haha this is an interesting thought. I'll explore a TTL of 1s to clients some more, although I can already think of a few reasons why this would be a bad idea for privacy.

though. Maybe dnsdist would be easier for this approach

For sure. And I plan to use dnsdist as the frontend anyway (also for load balancing/redundancy).

If you do not introduce delays in your cache hit respones it still will be trivial to tell them apart from cache hits. But introducing delays will create big queues of work in rec or dnsdist, so that has big drawbacks.

Introducing a fixed delay (that accommodates the time it takes for 99%+ backend lookup trips) before sending the record to the client actually is another mitigation that can be effective (at least on a conceptual level), but has some severe drawbacks indeed. Hence we try to focus on creating a big DNS cache where all records are effectively 'hot'.

Generally speaking, I believe what you are trying to achieve is hard.

I can imagine. What would be required to refresh cached records automatically? Would this require to change Recursor's code or is this something that can be done with Lua? I suspect this can only be done in C++. In that case, what would be the best method to implement these automated cache refreshes? I thought about two different options, but I don't know whether they will be viable or if there is a better method:

1. Iterate over the entire cache every x time (semi-random examples: 5 minutes for the whole cache or 1% chunks every 3 sec), check TTL against the threshold and if threshold for refresh has been met then trigger a refresh. The refresh mechanic maybe could be the same mechanic in use already for refreshing records with the 'hot cache' feature that gets triggered by a query for a cached record in Recursor.
2. Add another queue (separate from the queue used for the usual incoming requests) and when a record is cached, it's also proactively added to this queue for an upcoming refresh based on its TTL-x%.

Performance impact/Load wise, let's exclude the additional outgoing DNS queries, since that's another story and depends on many other factors such as minimal TTL as well. As far as I can tell the first option consists of fairly simple/efficient memory operations so shouldn't result in much additional load (<1%). To take it to the extreme: even 10M of these memory operations per second would be hard to notice on a modern server platform. I don't know how resource intensive the second options would be.

Thanks again for your insights! They really help.

[omoerbeek]

This is exactly the reason why as much records as possible should be cached. The higher the cache hit rate, the less likely the
earlier mentioned attacks can be successfully executed. 100% cache hit rate will be impossible, but it should be possible to get
close.

That is a bold statement, which I don't think is true. While it mightlook like that in your particular use case, one pseudo random attack on a wildcard name and the cache explodes.

As for refreshing: the current mechanism works as a best effort mechanism, with even some rate limiting built in. Using the same mechanism for a full cache refresh is not going to fly is my gut feeling. Not so much because it increases load of authoritative servers. The big pain point wil be the increase of the amount of work to be done by the recursor itself.

[sveeke]

That is a bold statement, which I don't think is true.

Yes I was still thinking for the past few days about this attack vector you mentioned before. That's a valid point! And haven't yet come up with a viable strategy (only very poor ones) to mitigate someone adding hundreds of millions domains that are resolvable but aren't useful (such as many randomized wildcard subdomains). Do you perhaps have a suggestion for a possible mitigation to this attack that I can explore further? Our infrastructure already gets attacked by certain governments that are a fan of our work frequently, and I wasn't planning on giving them another method to DoS our DNS infra.

[omoerbeek]

dnsdist has facilities that enable you to do dynamic blocking of (potential) malicious requests. See https://www.dnsdist.org/guides/dynblocks.html.