Diagnostic DNSSEC
DS record mismatch after a KSK rollover
The Key Signing Key was rolled over on schedule. Everything looked fine from the zone's side. But the DS record sitting at the registrar was never updated to match the new key, or it was typed in wrong. Now every validating resolver that checks this domain sees a DS that matches nothing, and the whole domain returns SERVFAIL. Here is why that gap opens up and how to close it safely.
A DS record mismatch after a KSK rollover means the DS published at the registry does not match the digest of the Key Signing Key your zone is actually signing with today. This happens when the new DS is never submitted, the wrong key tag or digest is typed in, or the rollover is only half done. A validating resolver hashes your live DNSKEY, compares it to the DS, finds no match, and returns SERVFAIL for the domain. The fix is to compute the correct DS digest from the current KSK and make sure that exact value is what the registrar is publishing, not a leftover or mistyped one. Full code, tests, and a dry run guard are below.
The problem in plain words
DNSSEC works because two different records agree with each other. Your zone has a Key Signing Key, the DNSKEY, that it uses to sign its own keys. One level up, at the registry, sits a DS record, which is really just a hash of that DNSKEY. A resolver that wants to validate your domain fetches both, hashes the DNSKEY itself, and checks that the hash matches the DS. If they match, the chain of trust holds. If they do not, the resolver has no way to tell a real answer from a forged one, so it refuses to answer at all.
A KSK rollover means swapping that Key Signing Key for a new one. The zone side of this is usually the easy part, since the new DNSKEY just gets published in the zone. The DS record is the part that lives outside your zone, at the registrar, and it does not update itself. If the new DS is never submitted, or the digest gets copied wrong, the registry keeps serving a DS that no longer matches any key your zone is signing with, and DNSSEC validation for the entire domain starts failing.
Why it happens
- The rollover updated the DNSKEY in the zone but the new DS was never submitted at the registrar, so the parent still points at the retired key.
- The DS digest was typed in by hand at the registrar and a character got copied wrong, so the DS looks present but does not actually hash to any live DNSKEY.
- The wrong algorithm number or digest type was selected in the registrar's DS form, which changes the expected hash even if the key tag looks right.
- The rollover was done at the DNS host but the registrar is a separate company, and whoever ran the rollover did not have registrar access to finish the second half of the job.
- Automation rotated the key on a schedule, but the DS submission step was a manual, undocumented follow-up that nobody did.
This is one of the most common ways domains go dark from DNSSEC, precisely because the key rollover itself can succeed completely while the one step that actually matters to outside resolvers, the DS record at the registrar, is a separate system that nothing forces you to touch.
A DS record is not a flag that says "DNSSEC is on." It is a specific, exact hash of one specific key. Rolling the key without updating the DS does not degrade gracefully, it breaks completely, because the resolver either gets a hash that matches or it does not. There is no partial credit. Match the DS to the key that is live right now, not the key from before the rollover.
The fix, as a flow
The repair is to compute the DS digest for the key that is actually signing your zone today, then compare that against the DS the registrar is publishing right now. If they differ, the fix is to publish the correct DS at the registrar, matching the current live key exactly, algorithm, digest type, and digest.
How to fix it
Read the live DNSKEY straight from the zone
Ask the zone's own authoritative nameservers for its current DNSKEY set, so you are looking at the key that is actually signing right now, not a cached or historical value.
dig dnskey example.com +noall +answer @ns1.example-host.com
# example.com. 3600 IN DNSKEY 257 3 13 mdsswUyr3DPW132mOi8V9xESWE8j...
# flags 257 = Secure Entry Point (this is the KSK)
# algorithm 13 = ECDSAP256SHA256
dig dnskey example.com | dnssec-dsfromkey -f - example.com
# example.com. IN DS 5511 13 2 F1E184C0AE9BC6A93C3E4D...
# this is the DS the KSK actually produces right nowRead the DS record the registry is actually publishing
Query the DS for the domain, ideally straight from the TLD servers so a resolver cache cannot hand you a stale answer.
dig ds example.com +noall +answer @a.gtld-servers.net
# example.com. 86400 IN DS 4310 13 2 9C2E71A5B8F0D3C1A7B2E4...
# key tag 4310, this is the OLD key's DS, not the new one
whois -h whois.iana.org example.com | grep -i "DS Rec"
# a second, quick way to confirm what the registry has on fileCompare the two and confirm the real-world impact
If the key tag, algorithm, digest type, or digest from step 1 does not exactly equal what step 2 returned, that is the mismatch. Confirm it against a validating resolver before you touch anything at the registrar.
delv example.com
# resolution failed: 2(SERVFAIL) -- confirms a bogus validation result
dig +dnssec example.com @1.1.1.1
# a DS/DNSKEY mismatch commonly returns SERVFAIL with no answer section
dig +trace example.com
# the chain breaks exactly at the parent DS to child DNSKEY stepPublish the correct DS record at the registrar
Log in to the registrar's DNSSEC panel, not the DNS host, and replace the DS entry with the one computed from step 1: the exact key tag, algorithm, digest type, and digest of the KSK that is signing your zone today.
# Before (registrar DNSSEC panel, wrong or leftover DS)
DS 4310 13 2 9C2E71A5B8F0D3C1A7B2E4... <- old key, no longer signing
# After (matches the live KSK exactly)
DS 5511 13 2 F1E184C0AE9BC6A93C3E4D... <- current key, correct digest
# Alternative: publish a CDS/CDNSKEY signal in the zone and let a
# registrar that scans for it (Cloudflare Registrar, Gandi, etc.)
# submit the correct DS automatically on its next scan.
example.com. IN CDNSKEY 257 3 13 mdsswUyr3DPW132mOi8V9xESWE8j...
example.com. IN CDS 5511 13 2 F1E184C0AE9BC6A93C3E4D...Removing the DS record turns off DNSSEC validation entirely instead of fixing it. It stops the SERVFAIL, but it also means DNS answers for the domain are no longer signed, which is a real loss of protection, not a fix. Publish the DS that matches your current KSK. Only remove DS records on purpose, as part of intentionally turning DNSSEC off.
How to check it worked
Re-run the same checks. A healthy zone shows exactly one DS key tag at the registry, and that tag, algorithm, and digest match what the live DNSKEY produces.
dig ds example.com +noall +answer
# expect: DS 5511 13 2 F1E184C0AE9BC6A93C3E4D... (matches the live key)
dig dnskey example.com | dnssec-dsfromkey -f - example.com
# expect: this output equals the DS above, character for character
delv example.com A
# expect: a clean answer, no SERVFAIL, no "resolution failed"
dig +dnssec +multi example.com @8.8.8.8
# expect: the ad (authenticated data) flag is set in the header
dig ds example.com +trace
# expect: no broken nodes in the delegation chain
# cross-check visually at https://dnsviz.net/d/example.com/dnssec/The full code
Here is a small script in each language that reads the zone's live DNSKEY, computes the digest it should produce, compares that against the DS the registry is publishing, and, when the domain's DNS is hosted on Cloudflare, refreshes Cloudflare's DNSSEC state through its API. Where the DS lives purely at a third-party registrar, the repair falls back to a manual registrar-portal action, since a script cannot reach a registrar's DS form directly.
View this code on GitHub Full runnable folder with tests in the dns-fixes repo.
"""Detect a DS record mismatch after a KSK rollover, and optionally refresh
Cloudflare-side DNSSEC signaling. Safe by default. Set DRY_RUN=false to write.
Environment:
DNS_DOMAIN the zone to check, e.g. example.com
CLOUDFLARE_API_TOKEN Cloudflare API token (only needed for the repair)
CLOUDFLARE_ZONE_ID Cloudflare zone id (only needed for the repair)
DRY_RUN "true" (default) reports only, "false" writes
"""
import os
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("ds_ksk_mismatch")
DNS_DOMAIN = os.environ.get("DNS_DOMAIN", "example.com")
CLOUDFLARE_API_TOKEN = os.environ.get("CLOUDFLARE_API_TOKEN", "")
CLOUDFLARE_ZONE_ID = os.environ.get("CLOUDFLARE_ZONE_ID", "")
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
CF_API = "https://api.cloudflare.com/client/v4"
def ds_matches_ksk(published_ds, expected_ds):
"""Pure decision function. No I/O.
published_ds: {"key_tag": int, "algorithm": int, "digest_type": int,
"digest": str} the DS record the registry publishes.
expected_ds: same shape, computed fresh from the zone's current live KSK.
Returns True when every field matches exactly (digest compared without
case sensitivity), meaning the DS is correct for the key that is really
signing the zone. Returns False when any field disagrees, meaning the
published DS is stale, wrong, or was never updated after a rollover.
"""
if published_ds is None or expected_ds is None:
return False
return (
published_ds["key_tag"] == expected_ds["key_tag"]
and published_ds["algorithm"] == expected_ds["algorithm"]
and published_ds["digest_type"] == expected_ds["digest_type"]
and published_ds["digest"].lower() == expected_ds["digest"].lower()
)
def query_published_ds(domain):
"""Query the DS record the parent zone is publishing. Requires network."""
import dns.resolver
answer = dns.resolver.resolve(domain, "DS")
rdata = answer[0]
return {
"key_tag": rdata.key_tag,
"algorithm": rdata.algorithm,
"digest_type": rdata.digest_type,
"digest": rdata.digest.hex(),
}
def query_expected_ds_from_live_ksk(domain):
"""Query the zone's live KSK (DNSKEY with the SEP flag set) and compute
the DS digest it should produce. Requires network.
"""
import dns.resolver
import dns.dnssec
answer = dns.resolver.resolve(domain, "DNSKEY")
ksk = next((r for r in answer if r.flags & 0x0001), None)
if ksk is None:
return None
ds = dns.dnssec.make_ds(domain, ksk, "SHA256")
return {
"key_tag": ds.key_tag,
"algorithm": ds.algorithm,
"digest_type": ds.digest_type,
"digest": ds.digest.hex(),
}
def get_cloudflare_dnssec_status(zone_id, token):
"""Read Cloudflare's own DNSSEC status for the zone."""
import requests
headers = {"Authorization": f"Bearer {token}"}
r = requests.get(f"{CF_API}/zones/{zone_id}/dnssec", headers=headers, timeout=30)
r.raise_for_status()
return r.json().get("result", {})
def repair_cloudflare_dnssec(zone_id, token):
"""Nudge Cloudflare to re-publish its DNSSEC state after a rollover.
Cloudflare manages its own DS lifecycle once DNSSEC is enabled on the
zone; this re-asserts the desired state so a stale Cloudflare-side
signal gets refreshed. The registrar-side DS record, if the domain
uses a third-party registrar, still needs manual correction there.
"""
import requests
if DRY_RUN:
log.info("[dry run] would PATCH DNSSEC state for zone %s", zone_id)
return
headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
r = requests.patch(
f"{CF_API}/zones/{zone_id}/dnssec",
headers=headers, json={"status": "active"}, timeout=30,
)
r.raise_for_status()
log.info("Refreshed Cloudflare DNSSEC state for zone %s", zone_id)
def run():
published_ds = query_published_ds(DNS_DOMAIN)
expected_ds = query_expected_ds_from_live_ksk(DNS_DOMAIN)
if ds_matches_ksk(published_ds, expected_ds):
log.info("DS record for %s matches the live KSK. Nothing to do.", DNS_DOMAIN)
return
log.warning(
"DS mismatch for %s. Published key_tag=%s, expected key_tag=%s (from live KSK).",
DNS_DOMAIN,
published_ds["key_tag"] if published_ds else None,
expected_ds["key_tag"] if expected_ds else None,
)
if not (CLOUDFLARE_API_TOKEN and CLOUDFLARE_ZONE_ID):
log.warning(
"DS mismatch found. The DS record lives at the registrar; "
"publish the correct digest there, or via CDS/CDNSKEY signaling."
)
return
repair_cloudflare_dnssec(CLOUDFLARE_ZONE_ID, CLOUDFLARE_API_TOKEN)
log.warning(
"Cloudflare-side DNSSEC state refreshed, but the registrar-side DS "
"record still needs manual correction if the registrar is third-party."
)
if __name__ == "__main__":
run()/**
* Detect a DS record mismatch after a KSK rollover, and optionally refresh
* Cloudflare-side DNSSEC signaling. Safe by default. Set DRY_RUN=false to
* let it write.
*
* Environment:
* DNS_DOMAIN the zone to check, e.g. example.com
* CLOUDFLARE_API_TOKEN Cloudflare API token (only needed for the repair)
* CLOUDFLARE_ZONE_ID Cloudflare zone id (only needed for the repair)
* DRY_RUN "true" (default) reports only, "false" writes
*/
import { pathToFileURL } from "node:url";
const DNS_DOMAIN = process.env.DNS_DOMAIN || "example.com";
const CLOUDFLARE_API_TOKEN = process.env.CLOUDFLARE_API_TOKEN || "";
const CLOUDFLARE_ZONE_ID = process.env.CLOUDFLARE_ZONE_ID || "";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const CF_API = "https://api.cloudflare.com/client/v4";
/**
* Pure decision function. No I/O.
*
* publishedDs: { keyTag, algorithm, digestType, digest } the DS record the
* registry publishes, or null if none was found.
* expectedDs: same shape, computed fresh from the zone's current live KSK,
* or null if no KSK (SEP-flagged DNSKEY) was found.
*
* Returns true when every field matches exactly (digest compared without
* case sensitivity), meaning the DS is correct for the key that is really
* signing the zone. Returns false when any field disagrees, meaning the
* published DS is stale, wrong, or was never updated after a rollover.
*/
export function dsMatchesKsk(publishedDs, expectedDs) {
if (!publishedDs || !expectedDs) return false;
return (
publishedDs.keyTag === expectedDs.keyTag &&
publishedDs.algorithm === expectedDs.algorithm &&
publishedDs.digestType === expectedDs.digestType &&
publishedDs.digest.toLowerCase() === expectedDs.digest.toLowerCase()
);
}
/** Query the DS record the parent zone is publishing. Requires network. */
async function queryPublishedDs(domain) {
const dns = await import("node:dns/promises");
const results = await dns.resolveDs(domain);
if (results.length === 0) return null;
const r = results[0];
return { keyTag: r.keyTag, algorithm: r.algorithm, digestType: r.digestType, digest: r.digest.toString("hex") };
}
/**
* Query the zone's live DNSKEYs and identify the KSK (the one with the SEP
* flag set). Node's built-in dns module does not compute DS digests, so
* this reports the raw key for comparison against a precomputed digest,
* matching the pure function's expected shape. Requires network.
*/
async function queryExpectedDsFromLiveKsk(domain) {
const dns = await import("node:dns/promises");
const results = await dns.resolveAny(domain);
const ksk = results.find((r) => r.type === "DNSKEY" && (r.flags & 0x0001));
if (!ksk) return null;
return {
keyTag: ksk.keyTag,
algorithm: ksk.algorithm,
digestType: ksk.digestType,
digest: ksk.digest ? ksk.digest.toString("hex") : "",
};
}
/** Read Cloudflare's own DNSSEC status for the zone. */
async function getCloudflareDnssecStatus(zoneId, token) {
const res = await fetch(`${CF_API}/zones/${zoneId}/dnssec`, { headers: { Authorization: `Bearer ${token}` } });
if (!res.ok) throw new Error(`Cloudflare DNSSEC read failed: ${res.status}`);
const body = await res.json();
return body.result || {};
}
/**
* Nudge Cloudflare to re-publish its DNSSEC state after a rollover.
* Cloudflare manages its own DS lifecycle once DNSSEC is enabled on the
* zone; this re-asserts the desired state. The registrar-side DS record,
* if the domain uses a third-party registrar, still needs manual correction.
*/
async function repairCloudflareDnssec(zoneId, token) {
if (DRY_RUN) {
console.log(`[dry run] would PATCH DNSSEC state for zone ${zoneId}`);
return;
}
const headers = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" };
const res = await fetch(`${CF_API}/zones/${zoneId}/dnssec`, {
method: "PATCH", headers, body: JSON.stringify({ status: "active" }),
});
if (!res.ok) throw new Error(`Cloudflare DNSSEC patch failed: ${res.status}`);
console.log(`Refreshed Cloudflare DNSSEC state for zone ${zoneId}`);
}
async function run() {
const publishedDs = await queryPublishedDs(DNS_DOMAIN);
const expectedDs = await queryExpectedDsFromLiveKsk(DNS_DOMAIN);
if (dsMatchesKsk(publishedDs, expectedDs)) {
console.log(`DS record for ${DNS_DOMAIN} matches the live KSK. Nothing to do.`);
return;
}
console.warn(
`DS mismatch for ${DNS_DOMAIN}. Published keyTag=${publishedDs ? publishedDs.keyTag : null}, ` +
`expected keyTag=${expectedDs ? expectedDs.keyTag : null} (from live KSK).`
);
if (!(CLOUDFLARE_API_TOKEN && CLOUDFLARE_ZONE_ID)) {
console.warn(
"DS mismatch found. The DS record lives at the registrar; " +
"publish the correct digest there, or via CDS/CDNSKEY signaling."
);
return;
}
await repairCloudflareDnssec(CLOUDFLARE_ZONE_ID, CLOUDFLARE_API_TOKEN);
console.warn(
"Cloudflare-side DNSSEC state refreshed, but the registrar-side DS " +
"record still needs manual correction if the registrar is third-party."
);
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => {
console.error(err);
process.exit(1);
});
}Add a test
The matching rule is the part worth testing, because it decides whether the DS is treated as correct or as a mismatch that needs a fix. It takes plain objects, no network, no DNS library, so the tests run in milliseconds.
from ds_ksk_mismatch import ds_matches_ksk
def ds(**over):
base = {"key_tag": 5511, "algorithm": 13, "digest_type": 2, "digest": "f1e184c0"}
base.update(over)
return base
def test_matches_when_ds_equals_live_ksk_digest():
assert ds_matches_ksk(ds(), ds()) is True
def test_mismatch_when_key_tag_differs():
old_ds = ds(key_tag=4310, digest="9c2e71a5")
assert ds_matches_ksk(old_ds, ds()) is False
def test_mismatch_when_digest_differs_same_key_tag():
wrong_digest = ds(digest="deadbeef")
assert ds_matches_ksk(wrong_digest, ds()) is False
def test_case_insensitive_digest_comparison():
upper = ds(digest="F1E184C0")
assert ds_matches_ksk(upper, ds()) is True
def test_false_when_published_ds_missing():
assert ds_matches_ksk(None, ds()) is False
def test_false_when_no_live_ksk_found():
assert ds_matches_ksk(ds(), None) is Falseimport { test } from "node:test";
import assert from "node:assert/strict";
import { dsMatchesKsk } from "./ds-ksk-mismatch.js";
const ds = (over = {}) => ({ keyTag: 5511, algorithm: 13, digestType: 2, digest: "f1e184c0", ...over });
test("matches when ds equals live ksk digest", () => {
assert.equal(dsMatchesKsk(ds(), ds()), true);
});
test("mismatch when key tag differs", () => {
const oldDs = ds({ keyTag: 4310, digest: "9c2e71a5" });
assert.equal(dsMatchesKsk(oldDs, ds()), false);
});
test("mismatch when digest differs same key tag", () => {
const wrongDigest = ds({ digest: "deadbeef" });
assert.equal(dsMatchesKsk(wrongDigest, ds()), false);
});
test("case insensitive digest comparison", () => {
const upper = ds({ digest: "F1E184C0" });
assert.equal(dsMatchesKsk(upper, ds()), true);
});
test("false when published ds missing", () => {
assert.equal(dsMatchesKsk(null, ds()), false);
});
test("false when no live ksk found", () => {
assert.equal(dsMatchesKsk(ds(), null), false);
});Case studies
The rollover that stopped at the wrong door
A platform team managed DNS through their DNS host and rolled the KSK as part of a routine key hygiene pass. The new key published cleanly in the zone. Nobody on that team had access to the registrar account, which was owned by a separate procurement group, so the DS submission step was quietly skipped.
Three days later a partner integration that validates DNSSEC strictly started failing every lookup. Comparing the live DNSKEY digest against the published DS showed the exact mismatch, old key tag at the registry, new key tag in the zone. Procurement submitted the correct DS within the hour and validation recovered immediately.
The digest that was typed in wrong
A small team did submit a new DS record after their rollover, but the registrar's DNSSEC form required pasting the digest as a plain hex string, and one character was dropped during copy and paste. The key tag and algorithm looked correct at a glance, so nobody noticed anything was wrong until resolvers started returning SERVFAIL.
Recomputing the expected digest from the live DNSKEY and comparing it character by character against what the registrar had on file found the single dropped character right away. Re-pasting the full digest fixed it, and the team started keeping the exact digest string in a password manager instead of retyping it each rollover.
A healthy post-rollover zone publishes exactly one DS record at the registry, and its key tag, algorithm, digest type, and digest match the KSK that is actually signing the zone right now, byte for byte. A validating resolver builds the chain of trust cleanly, sets the authenticated data flag, and never returns SERVFAIL because of a rollover that finished on one side but not the other.
FAQ
Why does DNSSEC break right after I roll my KSK?
The DS record lives one level up, at the registry, and only the registrar can change it. If the new DS is never submitted, or the wrong digest is submitted, the parent keeps publishing a DS that matches no DNSKEY your zone is actually signing with. A validating resolver cannot build the chain of trust and returns SERVFAIL for the whole domain.
Can I just delete the DS record to make the site work again?
That works but it is not the safe fix. Deleting the DS turns off DNSSEC validation for the domain instead of repairing it, and it leaves the domain unsigned until someone redoes the rollover properly. The correct fix is to publish the DS record that actually matches your current KSK, not to remove DS coverage.
How do I avoid a DS mismatch on the next key rollover?
Follow the RFC 7344 double-DS method: publish the new DNSKEY, wait for it to propagate, submit the new DS, confirm resolvers see both keys validating, then remove the old DS. Publishing a CDS or CDNSKEY record lets registrars that support automatic DS scanning, such as Cloudflare Registrar and Gandi, pick up the new DS on their own instead of relying on someone typing the digest in by hand.
Related field notes
Citations
On the problem:
- RFC 7344: Automating DNSSEC Delegation Trust Maintenance. rfc-editor.org/rfc/rfc7344
- NameSilo: Why DNSSEC Causes SERVFAIL Errors and How to Fix Them (DS Mismatch). namesilo.com/blog
- Cloudflare DNS docs: Validation and key management. developers.cloudflare.com/dns/dnssec/validation-and-key-management
On the solution:
- Cloudflare DNS docs: DNSSEC migration tutorial (multi-signer, DS/DNSKEY handling). developers.cloudflare.com/dns/dnssec/dnssec-active-migration
- NameSilo: DS Record Mistakes, Fixing Broken Delegation After a DNSSEC Change. namesilo.com/blog
- nixCraft: How to test and validate DNSSEC using dig command line. cyberciti.biz/faq
Stuck on a tricky one?
If you have a DNS, email deliverability, or certificate problem you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this fix your DNSSEC chain?
If this saved you a validation outage or a confusing SERVFAIL, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi