Diagnostic DNSSEC
Stale DS records left behind after a key rollover
A DNSSEC key rollover finished weeks ago. The new key works fine on its own. But someone forgot one small step at the registrar: delete the old DS record. Now resolvers see two DS entries, one that matches nothing in the zone, and validation quietly starts failing for anyone who checks DNSSEC. Here is why that old record keeps causing trouble and how to clean it up safely.
During a key rollover you publish a new DS record at the registrar but the old one never gets removed. The parent zone keeps serving both DS entries forever. A validating resolver sees the old DS key tag, looks for a DNSKEY in your zone that matches it, finds none, and cannot build a clean chain of trust, so it returns SERVFAIL for that name. The fix is to log in to the registrar (not the DNS host) and delete every DS record whose key tag has no matching, currently-signing DNSKEY, keeping only the DS that matches your live key. Full code, tests, and a dry run guard are below.
The problem in plain words
DNSSEC proves that your DNS answers were not tampered with, using a chain of signatures that runs from the root down to your zone. Part of that chain is a small record called DS, published by the registry one level up, that points at your zone's signing key, called the DNSKEY. Resolvers match the DS to the DNSKEY to prove the two agree.
When you rotate that signing key, the normal way to do it safely is to publish the new DS alongside the old one for a while, then remove the old DS once everyone has switched over. That last step, deleting the old DS at the registrar, is easy to forget, because the site itself keeps working. Nothing looks broken from a normal visit. But any resolver that tries to validate DNSSEC for your domain sees a DS record with a key tag that matches nothing in your zone, and that is enough to break the whole chain for anyone doing that check.
Why it happens
- The rollover checklist ends at "publish the new DS," and deleting the old one at the registrar is a separate, easy to skip, manual step.
- The DS record lives at the registrar, not at the DNS host, so a change made entirely inside the DNS provider's dashboard never touches it.
- Nothing appears broken for regular visitors, because most resolvers do not validate DNSSEC by default, so the problem hides until someone with a validating resolver, or a monitoring tool, hits the domain.
- The registrar's DNSSEC panel shows a flat list of DS records with no obvious flag for "this one is stale," so an old entry blends in with the current one.
This is a well known failure mode of manual DNSSEC key management, which is why RFC 7344 defines an automated way to signal DS changes to the parent using CDS and CDNSKEY records, so the registrar can pick up the change itself instead of relying on someone remembering a manual cleanup step.
A DS record is a promise: "the zone below has a DNSKEY that matches this exact key tag and digest." The registry does not check whether that promise is still true, it just keeps serving whatever DS records were entered until someone removes them. An old DS that no longer matches any live key is not harmless leftover data, it is an active claim that the resolver has to try to honor, and fails.
The fix, as a flow
The repair itself is simple: remove the DS record whose key tag has no matching DNSKEY, keeping only the DS that points at the key that is actually signing the zone right now. The part that takes care is making sure you are removing the right one, and not touching the chain while a resolver might still be caught mid-transition. Compare what the parent publishes against what the zone actually signs with, and only delete a DS once you are sure it is truly stale.
How to fix it
Pull the DS set the parent is publishing
Query the DS records for your domain, either from your normal resolver or straight from the TLD servers to bypass any caching. Note every key tag that comes back.
dig ds example.com +noall +answer
# example.com. 3600 IN DS 2371 13 2 3A1B9F... (current key)
# example.com. 3600 IN DS 55123 8 2 9C2E71... (old key, still published)
dig ds example.com @a.gtld-servers.net
# hits the TLD directly, bypassing any resolver cachePull the live DNSKEY set from the child zone
Ask the zone's own authoritative nameservers what keys it is actually signing with right now, then compute the DS digest each of those keys would produce.
dig dnskey example.com +noall +answer @ns1.example-host.com
dig dnskey example.com | dnssec-dsfromkey -f - example.com
# use -2 to only compute SHA-256 digests
# example.com. IN DS 2371 13 2 3A1B9F...
# only the live key tags print here, so 55123 is missing from this outputDiff the two lists and confirm the impact
Any DS key tag from step 1 that has no matching DNSKEY line in step 2 is stale. Confirm the real-world effect with a validating resolver before you touch anything.
dig +dnssec example.com @1.1.1.1
# a stale-DS mismatch commonly shows SERVFAIL with no answer
delv example.com
# resolution failed: 2(SERVFAIL) -- confirms a bogus validation result
dig +trace example.com
# the chain breaks right at the parent DS to child DNSKEY stepDelete the stale DS record at the registrar
Log in to the registrar's DNSSEC panel, not the DNS host, and remove the DS entry whose key tag had no match, leaving only the DS that matches the live key. If the registrar supports CDS or CDNSKEY scanning, publish a delete signal in the zone instead and let the registrar remove it automatically on its next scan.
# Before (registrar DNSSEC panel)
DS 2371 13 2 3A1B9F... <- current key, keep this one
DS 55123 8 2 9C2E71... <- stale, no matching DNSKEY, delete this one
# After
DS 2371 13 2 3A1B9F... <- only the current key remains
# Alternative: RFC 8078 delete signal published in the child zone,
# scanned automatically by supporting registrars (Cloudflare Registrar, Gandi, etc.)
example.com. IN CDNSKEY 0 3 0 AA==
example.com. IN CDS 0 0 0 00Always keep at least one valid DS record in place the entire time, per the RFC 7344/6781 double-DS rollover method. Wait for the old DS record's TTL to expire, typically about 1.5 times the published TTL of 3600 seconds, before deleting it, so no resolver is left holding a cached DS that no longer applies.
How to check it worked
Re-run the same queries. A clean domain returns exactly one DS key tag, and that tag has a matching DNSKEY, and end-to-end validation succeeds with the authenticated data flag set.
dig ds example.com +noall +answer
# expect: only the current key tag, e.g. DS 2371 13 2 3A1B9F...
dig dnskey example.com | dnssec-dsfromkey -f - example.com
# expect: every remaining DS key tag has a matching line here
delv example.com A
# expect: a clean answer, no SERVFAIL, no "resolution failed"
dig +dnssec +multi example.com @8.8.8.8
# expect: the ad (authenticated data) flag is set in the header
dig ds example.com +trace
# expect: no red/error nodes in the delegation chain
# cross-check visually at https://dnsviz.net/d/example.com/dnssec/The full code
Here is a small script in each language that pulls the DS set from the parent, computes the expected digest for every live DNSKEY, flags any stale DS records, and, when told to, cleans up Cloudflare-side DNSSEC signaling. Where the DS lives purely at a third-party registrar, the repair step falls back to a manual registrar-portal action.
View this code on GitHub Full runnable folder with tests in the dns-fixes repo.
"""Detect stale/orphaned DS records left behind after a DNSSEC key rollover,
and optionally repair Cloudflare-side DNSSEC signaling. Safe by default.
Set DRY_RUN=false to let it write.
"""
import os
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("stale_ds_records")
DNS_DOMAIN = os.environ.get("DNS_DOMAIN", "example.com")
CLOUDFLARE_API_TOKEN = os.environ.get("CLOUDFLARE_API_TOKEN", "")
CLOUDFLARE_ZONE_ID = os.environ.get("CLOUDFLARE_ZONE_ID", "")
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
CF_API = "https://api.cloudflare.com/client/v4"
def find_stale_ds(ds_records, dnskey_records):
"""Pure decision function. No I/O.
ds_records: [{"key_tag": int, "algorithm": int, "digest_type": int,
"digest": str}, ...] from the parent/registry
dnskey_records: [{"key_tag": int, "algorithm": int, "flags": int,
"digest": str}, ...] computed from the child zone's
live DNSKEYs (digest pre-hashed per digest_type)
Returns the subset of ds_records whose (key_tag, algorithm, digest) has
no matching entry in dnskey_records. Those are stale/orphaned DS
records that should be removed at the registrar.
"""
live = {
(k["key_tag"], k["algorithm"], k["digest"].lower())
for k in dnskey_records
}
stale = []
for ds in ds_records:
fingerprint = (ds["key_tag"], ds["algorithm"], ds["digest"].lower())
if fingerprint not in live:
stale.append(ds)
return stale
def query_parent_ds(domain):
"""Query the DS RRset the parent zone is publishing. Requires network."""
import dns.resolver
records = []
answer = dns.resolver.resolve(domain, "DS")
for rdata in answer:
records.append({
"key_tag": rdata.key_tag,
"algorithm": rdata.algorithm,
"digest_type": rdata.digest_type,
"digest": rdata.digest.hex(),
})
return records
def query_child_dnskeys_as_ds(domain):
"""Query the child zone's live DNSKEYs and compute the DS digest each
one would produce, so it can be compared against the parent's DS set.
Requires network.
"""
import dns.resolver
import dns.dnssec
import dns.rdatatype
records = []
answer = dns.resolver.resolve(domain, "DNSKEY")
for rdata in answer:
ds = dns.dnssec.make_ds(domain, rdata, "SHA256")
records.append({
"key_tag": ds.key_tag,
"algorithm": ds.algorithm,
"flags": rdata.flags,
"digest": ds.digest.hex(),
})
return records
def list_cloudflare_dnssec_ds(zone_id, token):
"""List DS records Cloudflare is publishing for its own DNSSEC signaling."""
import requests
headers = {"Authorization": f"Bearer {token}"}
url = f"{CF_API}/zones/{zone_id}/dnssec"
r = requests.get(url, headers=headers, timeout=30)
r.raise_for_status()
return r.json().get("result", {})
def repair_cloudflare_dnssec(zone_id, token):
"""Nudge Cloudflare to refresh its DNSSEC state after a rollover.
Cloudflare manages its own DS lifecycle once DNSSEC is enabled on the
zone; this simply re-asserts the desired state so any stale signaling
on the Cloudflare side is refreshed. The registrar-side DS record,
if the domain uses a third-party registrar, must be removed manually.
"""
import requests
if DRY_RUN:
log.info("[dry run] would PATCH DNSSEC state for zone %s", zone_id)
return
headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
url = f"{CF_API}/zones/{zone_id}/dnssec"
requests.patch(url, headers=headers, json={"status": "active"}, timeout=30).raise_for_status()
log.info("Refreshed Cloudflare DNSSEC state for zone %s", zone_id)
def run():
parent_ds = query_parent_ds(DNS_DOMAIN)
child_ds_equivalents = query_child_dnskeys_as_ds(DNS_DOMAIN)
stale = find_stale_ds(parent_ds, child_ds_equivalents)
if not stale:
log.info("No stale DS records found for %s.", DNS_DOMAIN)
return
for ds in stale:
log.warning(
"Stale DS found: key_tag=%s algorithm=%s digest_type=%s (no matching DNSKEY)",
ds["key_tag"], ds["algorithm"], ds["digest_type"],
)
if not (CLOUDFLARE_API_TOKEN and CLOUDFLARE_ZONE_ID):
log.warning(
"Stale DS records found. This zone's DS lives at the registrar; "
"remove it there manually, or via CDS/CDNSKEY delete signaling."
)
return
repair_cloudflare_dnssec(CLOUDFLARE_ZONE_ID, CLOUDFLARE_API_TOKEN)
log.warning(
"Cloudflare-side DNSSEC state refreshed, but the registrar-side DS "
"record still needs manual removal if the registrar is third-party."
)
if __name__ == "__main__":
run()/**
* Detect stale/orphaned DS records left behind after a DNSSEC key rollover,
* and optionally repair Cloudflare-side DNSSEC signaling. Safe by default.
* Set DRY_RUN=false to let it write.
*/
import { pathToFileURL } from "node:url";
const DNS_DOMAIN = process.env.DNS_DOMAIN || "example.com";
const CLOUDFLARE_API_TOKEN = process.env.CLOUDFLARE_API_TOKEN || "";
const CLOUDFLARE_ZONE_ID = process.env.CLOUDFLARE_ZONE_ID || "";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const CF_API = "https://api.cloudflare.com/client/v4";
/**
* Pure decision function. No I/O.
*
* dsRecords: [{ keyTag, algorithm, digestType, digest }, ...] from the
* parent/registry
* dnskeyRecords: [{ keyTag, algorithm, flags, digest }, ...] computed from
* the child zone's live DNSKEYs (digest pre-hashed per
* digestType for comparison)
*
* Returns the subset of dsRecords whose (keyTag, algorithm, digest) has no
* matching entry in dnskeyRecords. Those are stale/orphaned DS records
* that should be removed at the registrar.
*/
export function findStaleDs(dsRecords, dnskeyRecords) {
const live = new Set(
dnskeyRecords.map((k) => `${k.keyTag}:${k.algorithm}:${k.digest.toLowerCase()}`)
);
return dsRecords.filter((ds) => {
const fingerprint = `${ds.keyTag}:${ds.algorithm}:${ds.digest.toLowerCase()}`;
return !live.has(fingerprint);
});
}
/** Query the DS RRset the parent zone is publishing. Requires network. */
async function queryParentDs(domain) {
const dns = await import("node:dns/promises");
const results = await dns.resolveDs(domain);
return results.map((r) => ({
keyTag: r.keyTag,
algorithm: r.algorithm,
digestType: r.digestType,
digest: r.digest.toString("hex"),
}));
}
/**
* Query the child zone's live DNSKEYs. Node's built-in dns module does not
* compute DS digests, so this reports the raw DNSKEY set for comparison
* against a precomputed digest, matching the pure function's expected shape.
* Requires network.
*/
async function queryChildDnskeys(domain) {
const dns = await import("node:dns/promises");
const results = await dns.resolveAny(domain);
return results
.filter((r) => r.type === "DNSKEY")
.map((r) => ({
keyTag: r.keyTag,
algorithm: r.algorithm,
flags: r.flags,
digest: r.digest ? r.digest.toString("hex") : "",
}));
}
/** List DS records Cloudflare is publishing for its own DNSSEC signaling. */
async function listCloudflareDnssecDs(zoneId, token) {
const url = `${CF_API}/zones/${zoneId}/dnssec`;
const res = await fetch(url, { headers: { Authorization: `Bearer ${token}` } });
if (!res.ok) throw new Error(`Cloudflare DNSSEC read failed: ${res.status}`);
const body = await res.json();
return body.result || {};
}
/**
* Nudge Cloudflare to refresh its DNSSEC state after a rollover. Cloudflare
* manages its own DS lifecycle once DNSSEC is enabled on the zone; this
* simply re-asserts the desired state. The registrar-side DS record, if
* the domain uses a third-party registrar, must be removed manually.
*/
async function repairCloudflareDnssec(zoneId, token) {
if (DRY_RUN) {
console.log(`[dry run] would PATCH DNSSEC state for zone ${zoneId}`);
return;
}
const headers = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" };
const url = `${CF_API}/zones/${zoneId}/dnssec`;
const res = await fetch(url, { method: "PATCH", headers, body: JSON.stringify({ status: "active" }) });
if (!res.ok) throw new Error(`Cloudflare DNSSEC patch failed: ${res.status}`);
console.log(`Refreshed Cloudflare DNSSEC state for zone ${zoneId}`);
}
async function run() {
const parentDs = await queryParentDs(DNS_DOMAIN);
const childDnskeys = await queryChildDnskeys(DNS_DOMAIN);
const stale = findStaleDs(parentDs, childDnskeys);
if (stale.length === 0) {
console.log(`No stale DS records found for ${DNS_DOMAIN}.`);
return;
}
for (const ds of stale) {
console.warn(
`Stale DS found: keyTag=${ds.keyTag} algorithm=${ds.algorithm} digestType=${ds.digestType} (no matching DNSKEY)`
);
}
if (!(CLOUDFLARE_API_TOKEN && CLOUDFLARE_ZONE_ID)) {
console.warn(
"Stale DS records found. This zone's DS lives at the registrar; " +
"remove it there manually, or via CDS/CDNSKEY delete signaling."
);
return;
}
await repairCloudflareDnssec(CLOUDFLARE_ZONE_ID, CLOUDFLARE_API_TOKEN);
console.warn(
"Cloudflare-side DNSSEC state refreshed, but the registrar-side DS " +
"record still needs manual removal if the registrar is third-party."
);
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => {
console.error(err);
process.exit(1);
});
}Add a test
The matching rule is the part worth testing, because it decides which DS records get flagged as stale. It takes plain lists of dictionaries, no network, no DNS lookups, so the tests run in milliseconds.
from stale_ds_records import find_stale_ds
def ds(**over):
base = {"key_tag": 2371, "algorithm": 13, "digest_type": 2, "digest": "3a1b9f"}
base.update(over)
return base
def dnskey(**over):
base = {"key_tag": 2371, "algorithm": 13, "flags": 257, "digest": "3a1b9f"}
base.update(over)
return base
def test_no_stale_when_ds_matches_dnskey():
assert find_stale_ds([ds()], [dnskey()]) == []
def test_flags_ds_with_no_matching_dnskey():
old_ds = ds(key_tag=55123, algorithm=8, digest="9c2e71")
result = find_stale_ds([ds(), old_ds], [dnskey()])
assert result == [old_ds]
def test_flags_all_when_no_dnskeys_present():
old_ds = ds()
assert find_stale_ds([old_ds], []) == [old_ds]
def test_case_insensitive_digest_comparison():
upper_ds = ds(digest="3A1B9F")
assert find_stale_ds([upper_ds], [dnskey()]) == []import { test } from "node:test";
import assert from "node:assert/strict";
import { findStaleDs } from "./stale-ds-records.js";
const ds = (over = {}) => ({ keyTag: 2371, algorithm: 13, digestType: 2, digest: "3a1b9f", ...over });
const dnskey = (over = {}) => ({ keyTag: 2371, algorithm: 13, flags: 257, digest: "3a1b9f", ...over });
test("no stale when ds matches dnskey", () => {
assert.deepEqual(findStaleDs([ds()], [dnskey()]), []);
});
test("flags ds with no matching dnskey", () => {
const oldDs = ds({ keyTag: 55123, algorithm: 8, digest: "9c2e71" });
const result = findStaleDs([ds(), oldDs], [dnskey()]);
assert.deepEqual(result, [oldDs]);
});
test("flags all when no dnskeys present", () => {
const oldDs = ds();
assert.deepEqual(findStaleDs([oldDs], []), [oldDs]);
});
test("case insensitive digest comparison", () => {
const upperDs = ds({ digest: "3A1B9F" });
assert.deepEqual(findStaleDs([upperDs], [dnskey()]), []);
});Case studies
The transfer that dragged an old DS record along
A team rotated their DNSSEC key at their DNS host and confirmed the site worked fine. Months later they transferred the domain to a new registrar, and the transfer tool copied over every DS record it found, including the one from before the rollover that nobody had removed. The new registrar happily kept publishing both.
Nothing looked wrong until a partner's monitoring system, which does validate DNSSEC, started reporting SERVFAIL for the domain. Pulling the DS set and diffing it against the live DNSKEY showed the leftover key tag immediately, and deleting it at the new registrar's DNSSEC panel fixed it within the hour.
The rollover checklist that stopped one step early
An internal runbook for DNSSEC key rollovers listed generate key, publish DNSKEY, submit new DS, verify resolution. It never mentioned removing the old DS once the new one was confirmed. Every rollover since then left one more stale DS record sitting at the registrar, and after three rollovers the DNSSEC panel showed three DS entries where only one was ever needed.
Running the digest comparison against the live DNSKEY flagged two of the three as stale in seconds. The team removed both, kept the current one, and added the missing step back into the runbook.
A healthy, rolled-over zone publishes exactly one DS record set at the parent, and every key tag in it has a matching, currently-signing DNSKEY in the child zone. A validating resolver builds the chain of trust cleanly, gets the authenticated data flag set, and never returns SERVFAIL because of a leftover key from a rollover that finished long ago.
FAQ
Why does my domain fail DNSSEC validation after I rotated my DNSSEC key?
The registrar is still publishing the old DS record from before the rollover, alongside the new one. A validating resolver sees a DS key tag with no matching, currently-signing DNSKEY in your zone and cannot build a clean chain of trust, so it returns SERVFAIL or marks the zone bogus.
Is it safe to delete an old DS record myself?
Yes, as long as you first confirm the remaining DS record matches a DNSKEY that is actually live and signing in your zone right now. Never delete every DS at once mid-rollover. Keep at least one valid DS in place the whole time, and wait roughly 1.5 times the old record's TTL before removing it so no resolver is caught mid-cache.
How do I avoid this happening on the next key rollover?
Use the RFC 7344 double-DS rollover method and publish a CDS or CDNSKEY delete signal in your zone once the new key is live. Registrars that scan for CDS/CDNSKEY, including Cloudflare Registrar and Gandi, will then remove the old DS automatically on their next scan, so nobody has to remember a manual cleanup step.
Related field notes
Citations
On the problem:
- RFC 7344: Automating DNSSEC Delegation Trust Maintenance. rfc-editor.org/rfc/rfc7344
- NameSilo: Why DNSSEC Causes SERVFAIL Errors and How to Fix Them (DS Mismatch). namesilo.com/blog
- Cloudflare DNS docs: Validation and key management. developers.cloudflare.com/dns/dnssec/validation-and-key-management
On the solution:
- Cloudflare DNS docs: DNSSEC migration tutorial (multi-signer, DS/DNSKEY handling). developers.cloudflare.com/dns/dnssec/dnssec-active-migration
- NameSilo: DS Record Mistakes, Fixing Broken Delegation After a DNSSEC Change. namesilo.com/blog
- nixCraft: How to test and validate DNSSEC using dig command line. cyberciti.biz/faq
Stuck on a tricky one?
If you have a DNS, email deliverability, or certificate problem you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this fix your DNSSEC chain?
If this saved you a validation outage or a confusing SERVFAIL, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi