Reconciler Provider API / Reconciliation
Stale ACME challenge TXT record blocks renewal
A certificate renewal fails with an error like "incorrect TXT record" or "domain: unauthorized," even though your ACME client just published a fresh challenge. The usual cause is not a broken renewal at all. It is an old _acme-challenge TXT record from a prior failed attempt that never got cleaned up, still sitting in the zone, still a valid answer to anyone who asks. Here is why that happens and a script that finds the leftover and removes it.
ACME DNS-01 validation works by publishing a one-time token in a TXT record at _acme-challenge.example.com, letting the certificate authority read it, then deleting it. If the cleanup step fails, crashes, or gets skipped, that old token stays in the zone. On the next renewal, the record set has more than one value, and if the resolver returns the stale one, or nameservers disagree, validation fails. Run dig +short TXT _acme-challenge.example.com, delete every value that is not the token your client just published, and add a small script that clears out anything older than about an hour so this cannot pile up again. Full code, tests, and a dry run guard are below.
The problem in plain words
DNS-01 validation is a short conversation. Your ACME client, such as certbot, asks the certificate authority for a challenge. It publishes the answer as a TXT record at _acme-challenge.example.com. The certificate authority looks up that name, checks the value matches what it expects, and issues the certificate. Afterward, the client is supposed to delete that TXT record, because its job is done.
The trouble is that last step, the delete, is a separate action from the publish, usually run by a hook script. If that hook fails, times out, or the whole renewal process gets killed partway through, the delete never happens. The record just sits there. Nothing about DNS forces it to expire on its own unless you set a very short TTL and even then a resolver may still hold onto the old answer for a while. Most DNS provider APIs also add records rather than replacing them in place, so an old token and a new token can both exist at the same name at once.
Why it happens
- A certbot
--manual-cleanup-hook, or the equivalent hook in another ACME client, only runs when the earlier steps succeed. If validation itself failed or the process was killed, the hook that deletes the record may never fire. - RFC 8555 DNS-01 validation only needs one matching TXT value at the name, so most ACME clients and DNS provider APIs add a new record rather than replacing the old one in place. Nothing in the protocol forces a provider to clean up after itself.
- Wildcard and multi-domain certificates request several challenges at once, each writing to the same
_acme-challengename, or a subdomain variant of it, which makes an accidental pile-up of leftover values more likely. - Automation retries a failed renewal on a fixed schedule without checking whether the DNS side was left clean first, so a stale record from attempt one is still there when attempt two runs.
This is reported often on both the Let's Encrypt community forum and DNS provider community forums, usually phrased as "it worked last time, now it just fails" right after a renewal that had some kind of hiccup. See the citations at the end for the exact threads.
A DNS-01 challenge is only ever valid for a few minutes, the length of a single renewal attempt. Any _acme-challenge TXT record still standing an hour later is not part of anything currently happening. It is safe to treat it as leftover and delete it, because a real, in-progress validation would already be finished by then.
The fix, as a flow
List every TXT record at _acme-challenge.example.com through the DNS provider's API, check each one's age, and delete anything older than a short timeout, such as one hour, well past how long a real challenge ever takes. If a validation is currently in flight, its token is the one that is very recent and matches what the ACME client just published, so it is left alone. Everything else gets cleared before the CA looks at the record.
How to fix it
Look at what is actually published right now
Query the challenge name directly and see how many values come back. A healthy state during a renewal shows exactly one TXT value, matching the token the ACME client just requested. Two or more values, or one that does not match what the client just tried to set, points at a stale record.
dig +short TXT _acme-challenge.example.com
# a bad result looks like this: two values, one clearly older
# "old-token-from-a-failed-run-weeks-ago"
# "fresh-token-the-client-just-published"
# check every authoritative nameserver, not just one cache
dig NS example.com +short
dig @ns1.example.com TXT _acme-challenge.example.com +short
dig @ns2.example.com TXT _acme-challenge.example.com +short
# cross-check against a public resolver too
dig @8.8.8.8 TXT _acme-challenge.example.com +shortRead the exact error from the ACME client
The certbot or ACME client log usually names the leftover value directly. Look for a line naming the value the CA found, which is the old token, next to the value your client just published.
Incorrect TXT record "old-token-from-a-failed-run" found at
_acme-challenge.example.com
Domain: example.com
Type: unauthorized
Detail: ... did not receive a certificate ...List every record at the challenge name via the DNS provider API
If you manage the zone through Cloudflare, list the TXT records at that exact name and look at the modified_on timestamp on each one. A bad result is more than one record, or a record whose timestamp predates your current renewal attempt by more than a few minutes.
curl -s -H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=TXT&name=_acme-challenge.example.com" \
| jq '.result[] | {id, content, modified_on}'
# a bad result is more than one item in result, or a modified_on
# that is minutes or hours older than when your client just publishedDelete the stale record, keep only the current one
Delete every TXT record at that name whose value does not match the token your ACME client just generated. In a manual or registrar UI, open the zone editor, find the _acme-challenge TXT records, and remove the ones that are not current. Through the Cloudflare API, delete by record ID.
# delete each stale record by its id, one call per record
curl -s -X DELETE \
-H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID"
# the final TXT set for _acme-challenge.example.com should hold
# only the current challenge value, or nothing at all once the
# ACME client's own cleanup step runs after issuanceFix the root cause in the ACME client too
Make sure the cleanup hook actually runs even when validation itself fails, by wrapping it in a finally block or a shell trap so it always fires. That alone will not catch every crash, so pair it with the scheduled reconciliation script below.
#!/usr/bin/env bash
# wrap certbot's own hook call so cleanup always runs,
# even if validation itself failed partway through
set -e
trap 'bash /etc/letsencrypt/cleanup-hook.sh' EXIT
certbot renew \
--manual \
--preferred-challenges dns \
--manual-auth-hook /etc/letsencrypt/auth-hook.sh \
--manual-cleanup-hook /etc/letsencrypt/cleanup-hook.shHow to check it worked
Re-query the record, confirm it is a single clean value or empty, then re-run the renewal for real and watch it finish without the unauthorized error.
# should return exactly one value, or nothing at all
dig +short TXT _acme-challenge.example.com
# confirm every authoritative nameserver agrees
for ns in $(dig NS example.com +short); do
echo "$ns:"
dig @"$ns" TXT _acme-challenge.example.com +short
done
# dry run the renewal first, then run it for real
certbot renew --cert-name example.com --dry-run
certbot renew --cert-name example.com
# confirm the new certificate is live with a fresh expiry date
openssl s_client -connect example.com:443 -servername example.com \
</dev/null 2>/dev/null | openssl x509 -noout -datesA good result is a single matching TXT value (or none) that every nameserver agrees on, a renewal that completes with no "incorrect TXT record" or "unauthorized" error, and a certificate whose notAfter date is roughly ninety days out, which is the Let's Encrypt default.
The full code
Here is the complete checker and repair script in one file for each language. It reads the live TXT records at the challenge name, flags anything older than a configurable timeout or that does not match the token currently in flight, and, only when you turn dry run off, deletes those stale records through the Cloudflare API.
View this code on GitHub Full runnable folder with tests in the dns-fixes repo.
"""Detect a stale _acme-challenge TXT record left behind by a failed
ACME DNS-01 renewal, and optionally repair the zone via Cloudflare by
deleting the stale record(s).
Safe by default. Set DRY_RUN=false to let it write.
Env vars:
DNS_DOMAIN the domain to check, e.g. "example.com"
CURRENT_CHALLENGE_TOKEN the token the ACME client is currently
trying to validate, if any (leave unset or
empty when no validation is in flight)
STALE_TIMEOUT_SECONDS default 3600; age past which a record is
always considered stale
CLOUDFLARE_API_TOKEN Cloudflare API token (only needed for repair)
CLOUDFLARE_ZONE_ID Cloudflare zone id (only needed for repair)
DRY_RUN default "true"; set to "false" to actually write
"""
import os
import time
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("stale_acme_challenge_record")
DNS_DOMAIN = os.environ.get("DNS_DOMAIN", "example.com")
CURRENT_CHALLENGE_TOKEN = os.environ.get("CURRENT_CHALLENGE_TOKEN") or None
STALE_TIMEOUT_SECONDS = int(os.environ.get("STALE_TIMEOUT_SECONDS", "3600"))
CLOUDFLARE_API_TOKEN = os.environ.get("CLOUDFLARE_API_TOKEN", "")
CLOUDFLARE_ZONE_ID = os.environ.get("CLOUDFLARE_ZONE_ID", "")
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
CF_API = "https://api.cloudflare.com/client/v4"
GRACE_SECONDS = 300
def find_stale_challenge_records(records, current_token, now_ts, timeout_s=3600):
"""Pure decision function. No I/O.
records: list of TXT record dicts for _acme-challenge.<domain>, each
with 'id', 'content', and 'modified_on' as epoch seconds.
current_token: the token the ACME client is currently trying to
validate, or None if no validation is in flight.
now_ts: the current time as epoch seconds.
timeout_s: age in seconds past which a record is always stale,
regardless of its content, since a real challenge never
takes anywhere near this long.
Returns the list of record ids that are stale: any record older
than timeout_s, or, when current_token is not None, any record
whose content differs from current_token once it is older than a
short grace period (so a record published moments ago as part of
the in-flight challenge is never flagged before it has had a
chance to be read by the CA).
"""
stale_ids = []
for record in records:
age = now_ts - record["modified_on"]
if age > timeout_s:
stale_ids.append(record["id"])
continue
if current_token is not None and record["content"] != current_token and age > GRACE_SECONDS:
stale_ids.append(record["id"])
return stale_ids
def list_challenge_records(domain):
"""List every TXT record at _acme-challenge.<domain> via Cloudflare."""
import requests
headers = {"Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}"}
params = {"type": "TXT", "name": f"_acme-challenge.{domain}", "per_page": 100}
r = requests.get(
f"{CF_API}/zones/{CLOUDFLARE_ZONE_ID}/dns_records",
headers=headers, params=params, timeout=30,
)
r.raise_for_status()
return r.json()["result"]
def delete_record(record_id):
import requests
headers = {"Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}"}
url = f"{CF_API}/zones/{CLOUDFLARE_ZONE_ID}/dns_records/{record_id}"
if DRY_RUN:
log.info("[dry run] would delete record %s", record_id)
return
requests.delete(url, headers=headers, timeout=30).raise_for_status()
log.info("Deleted record %s", record_id)
def run():
records = list_challenge_records(DNS_DOMAIN)
if not records:
log.info("No _acme-challenge TXT records found for %s.", DNS_DOMAIN)
return
now_ts = int(time.time())
stale_ids = find_stale_challenge_records(records, CURRENT_CHALLENGE_TOKEN, now_ts, STALE_TIMEOUT_SECONDS)
if not stale_ids:
log.info("All %d TXT record(s) at _acme-challenge.%s look current.", len(records), DNS_DOMAIN)
return
for record_id in stale_ids:
log.warning("Stale _acme-challenge TXT record for %s: id %s", DNS_DOMAIN, record_id)
if not (CLOUDFLARE_API_TOKEN and CLOUDFLARE_ZONE_ID):
log.warning("No Cloudflare credentials set. Not repairing, only reporting.")
return
for record_id in stale_ids:
delete_record(record_id)
log.info("Done. %d stale record(s) %s.", len(stale_ids), "would be removed" if DRY_RUN else "removed")
if __name__ == "__main__":
run()/**
* Detect a stale _acme-challenge TXT record left behind by a failed
* ACME DNS-01 renewal, and optionally repair the zone via Cloudflare
* by deleting the stale record(s).
*
* Safe by default. Set DRY_RUN=false to let it write.
*
* Env vars:
* DNS_DOMAIN the domain to check, e.g. "example.com"
* CURRENT_CHALLENGE_TOKEN the token the ACME client is currently
* trying to validate, if any (leave unset
* or empty when no validation is in flight)
* STALE_TIMEOUT_SECONDS default 3600; age past which a record is
* always considered stale
* CLOUDFLARE_API_TOKEN Cloudflare API token (only needed for repair)
* CLOUDFLARE_ZONE_ID Cloudflare zone id (only needed for repair)
* DRY_RUN default "true"; set to "false" to actually write
*/
import { pathToFileURL } from "node:url";
const DNS_DOMAIN = process.env.DNS_DOMAIN || "example.com";
const CURRENT_CHALLENGE_TOKEN = process.env.CURRENT_CHALLENGE_TOKEN || null;
const STALE_TIMEOUT_SECONDS = Number(process.env.STALE_TIMEOUT_SECONDS || 3600);
const CLOUDFLARE_API_TOKEN = process.env.CLOUDFLARE_API_TOKEN || "";
const CLOUDFLARE_ZONE_ID = process.env.CLOUDFLARE_ZONE_ID || "";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const CF_API = "https://api.cloudflare.com/client/v4";
const GRACE_SECONDS = 300;
export function findStaleChallengeRecords(records, currentToken, nowTs, timeoutS = 3600) {
// Pure decision function. No I/O.
//
// records: list of TXT record objects for _acme-challenge.<domain>,
// each with 'id', 'content', and 'modified_on' as epoch seconds.
// currentToken: the token the ACME client is currently trying to
// validate, or null if no validation is in flight.
// nowTs: the current time as epoch seconds.
// timeoutS: age in seconds past which a record is always stale,
// regardless of its content, since a real challenge never takes
// anywhere near this long.
//
// Returns the list of record ids that are stale: any record older
// than timeoutS, or, when currentToken is not null, any record
// whose content differs from currentToken once it is older than a
// short grace period.
const staleIds = [];
for (const record of records) {
const age = nowTs - record.modified_on;
if (age > timeoutS) {
staleIds.push(record.id);
continue;
}
if (currentToken !== null && record.content !== currentToken && age > GRACE_SECONDS) {
staleIds.push(record.id);
}
}
return staleIds;
}
async function listChallengeRecords(domain) {
const headers = { Authorization: `Bearer ${CLOUDFLARE_API_TOKEN}` };
const params = new URLSearchParams({ type: "TXT", name: `_acme-challenge.${domain}`, per_page: "100" });
const res = await fetch(
`${CF_API}/zones/${CLOUDFLARE_ZONE_ID}/dns_records?${params.toString()}`,
{ headers },
);
if (!res.ok) throw new Error(`Cloudflare list returned ${res.status}`);
const body = await res.json();
return body.result;
}
async function deleteRecord(recordId) {
const headers = { Authorization: `Bearer ${CLOUDFLARE_API_TOKEN}` };
if (DRY_RUN) {
console.log(`[dry run] would delete record ${recordId}`);
return;
}
const res = await fetch(`${CF_API}/zones/${CLOUDFLARE_ZONE_ID}/dns_records/${recordId}`, {
method: "DELETE",
headers,
});
if (!res.ok) throw new Error(`Cloudflare delete returned ${res.status}`);
console.log(`Deleted record ${recordId}`);
}
async function run() {
const records = await listChallengeRecords(DNS_DOMAIN);
if (records.length === 0) {
console.log(`No _acme-challenge TXT records found for ${DNS_DOMAIN}.`);
return;
}
const nowTs = Math.floor(Date.now() / 1000);
const staleIds = findStaleChallengeRecords(records, CURRENT_CHALLENGE_TOKEN, nowTs, STALE_TIMEOUT_SECONDS);
if (staleIds.length === 0) {
console.log(`All ${records.length} TXT record(s) at _acme-challenge.${DNS_DOMAIN} look current.`);
return;
}
for (const recordId of staleIds) {
console.warn(`Stale _acme-challenge TXT record for ${DNS_DOMAIN}: id ${recordId}`);
}
if (!(CLOUDFLARE_API_TOKEN && CLOUDFLARE_ZONE_ID)) {
console.warn("No Cloudflare credentials set. Not repairing, only reporting.");
return;
}
for (const recordId of staleIds) {
await deleteRecord(recordId);
}
console.log(`Done. ${staleIds.length} stale record(s) ${DRY_RUN ? "would be removed" : "removed"}.`);
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => {
console.error(err);
process.exit(1);
});
}Add a test
The decision rule is the part most worth testing, because it decides which records are safe to delete. Because find_stale_challenge_records is pure, the test needs no network and no DNS provider. It just feeds in plain dicts and timestamps and checks which ids come back as stale.
from stale_acme_challenge_record import find_stale_challenge_records
NOW = 1_700_000_000
def record(id_, content, age_seconds):
return {"id": id_, "content": content, "modified_on": NOW - age_seconds}
def test_no_stale_when_single_fresh_record_matches_token():
records = [record("r1", "fresh-token", 30)]
assert find_stale_challenge_records(records, "fresh-token", NOW) == []
def test_flags_record_older_than_timeout_regardless_of_content():
records = [record("r1", "old-token", 7200)]
assert find_stale_challenge_records(records, None, NOW, timeout_s=3600) == ["r1"]
def test_flags_mismatched_token_past_grace_period():
records = [record("r1", "old-token", 600)]
assert find_stale_challenge_records(records, "fresh-token", NOW) == ["r1"]
def test_does_not_flag_mismatched_token_within_grace_period():
records = [record("r1", "old-token", 60)]
assert find_stale_challenge_records(records, "fresh-token", NOW) == []
def test_mixed_set_flags_only_the_stale_one():
records = [record("r1", "fresh-token", 30), record("r2", "old-token", 10800)]
assert find_stale_challenge_records(records, "fresh-token", NOW) == ["r2"]
def test_no_current_token_only_uses_timeout():
records = [record("r1", "anything", 300)]
assert find_stale_challenge_records(records, None, NOW, timeout_s=3600) == []
def test_empty_records_returns_empty_list():
assert find_stale_challenge_records([], "fresh-token", NOW) == []import { test } from "node:test";
import assert from "node:assert/strict";
import { findStaleChallengeRecords } from "./stale-acme-challenge-record.js";
const NOW = 1_700_000_000;
function record(id, content, ageSeconds) {
return { id, content, modified_on: NOW - ageSeconds };
}
test("no stale when single fresh record matches token", () => {
const records = [record("r1", "fresh-token", 30)];
assert.deepEqual(findStaleChallengeRecords(records, "fresh-token", NOW), []);
});
test("flags record older than timeout regardless of content", () => {
const records = [record("r1", "old-token", 7200)];
assert.deepEqual(findStaleChallengeRecords(records, null, NOW, 3600), ["r1"]);
});
test("flags mismatched token past grace period", () => {
const records = [record("r1", "old-token", 600)];
assert.deepEqual(findStaleChallengeRecords(records, "fresh-token", NOW), ["r1"]);
});
test("does not flag mismatched token within grace period", () => {
const records = [record("r1", "old-token", 60)];
assert.deepEqual(findStaleChallengeRecords(records, "fresh-token", NOW), []);
});
test("mixed set flags only the stale one", () => {
const records = [record("r1", "fresh-token", 30), record("r2", "old-token", 10800)];
assert.deepEqual(findStaleChallengeRecords(records, "fresh-token", NOW), ["r2"]);
});
test("no current token only uses timeout", () => {
const records = [record("r1", "anything", 300)];
assert.deepEqual(findStaleChallengeRecords(records, null, NOW, 3600), []);
});
test("empty records returns empty list", () => {
assert.deepEqual(findStaleChallengeRecords([], "fresh-token", NOW), []);
});Case studies
The renewal that failed once and then failed forever
A team ran a wildcard certificate renewal through a shared CI runner. One run got interrupted when the runner was killed for hitting a time limit, right after the challenge TXT record was published but before the cleanup hook ran. Every renewal after that failed with an "incorrect TXT record" error, because the old token was still sitting there next to each new one.
Nobody thought to check the DNS zone itself, since the error read like an ACME problem. A single dig +short TXT _acme-challenge.example.com showed two values, and deleting the old one fixed the very next renewal attempt.
Four subdomains, one broken cleanup script
A certificate covering four subdomains ran four separate DNS-01 challenges. A bug in a custom cleanup script meant it only ever cleaned up the first challenge name it was given and silently ignored the rest. For weeks, three of the four _acme-challenge names quietly accumulated a new leftover TXT record on every renewal.
Renewal still worked for a while, since the CA read the newest matching value most of the time, until nameserver disagreement finally made one of them serve a stale answer during validation. A scheduled script that deletes anything older than an hour would have caught this on day one instead of week three.
After the fix, dig +short TXT _acme-challenge.example.com returns exactly one value during a renewal, and nothing at all once the ACME client's own cleanup finishes. Every authoritative nameserver agrees. Renewal completes without an unauthorized or incorrect TXT record error, and the certificate's notAfter date rolls forward every time, on schedule, with no manual intervention.
FAQ
Why does an old _acme-challenge TXT record break certificate renewal?
DNS-01 validation checks the _acme-challenge TXT record for the exact token the certificate authority just issued. If a cleanup step failed or was skipped on an earlier attempt, the old token is still sitting there. The name now holds more than one value, and if the resolver returns the stale one first, or the provider disagrees across nameservers, validation fails with an unauthorized or incorrect TXT record error.
Is it safe to just delete every _acme-challenge TXT record I find?
It is safe once you confirm no validation is currently in flight, or once the record is older than a real challenge ever needs to be, typically a few minutes at most. A record older than about an hour is not part of an active renewal. Deleting it cannot break a validation that already finished or that has not started yet.
How do I stop this from happening again on future renewals?
Make sure your ACME client's cleanup hook actually runs even when validation fails, by wrapping it so cleanup always executes. On top of that, add a small scheduled script that lists every _acme-challenge TXT record, checks its age, and deletes anything older than about an hour, so an orphaned record never has the chance to sit there until the next renewal.
Related field notes
Citations
On the problem:
- Let's Encrypt Community Support: _acme-challenge Incorrect TXT record, old data. community.letsencrypt.org
- Cloudflare Community: extra ACME TXT records preventing renewal. community.cloudflare.com
- RFC 8555: Automatic Certificate Management Environment (ACME), section 8.4, DNS-01. datatracker.ietf.org/doc/html/rfc8555
On the solution:
- Cloudflare API docs: DNS Records, list. developers.cloudflare.com/api
- Cloudflare API docs: DNS Records, delete. developers.cloudflare.com/api
- Certbot User Guide: manual-auth-hook and manual-cleanup-hook. eff-certbot.readthedocs.io
Stuck on a tricky one?
If you have a DNS, domain, or certificate problem you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this fix your renewal?
If this got your certificate renewing cleanly again, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi