Reconciler Provider API / Reconciliation
DNS-managed record overwritten without an ownership check
A record that a person created by hand, or that another team's automation owns, gets quietly changed or deleted by a tool like ExternalDNS or a Terraform provider. Nobody asked for the change. Nobody reassigned the record on purpose. The reconciler just diffed its desired state against the live zone and wrote over whatever did not match, because it never checked who was actually allowed to touch that name first.
Reconcilers like ExternalDNS and Terraform manage a zone by comparing a desired state to what is live, then writing whatever is different. If the tool never checks an ownership marker before writing, it can overwrite or delete a record created by a human, a different team's automation, or a different tool entirely. ExternalDNS's own TXT registry is supposed to prevent this by storing heritage=external-dns,external-dns/owner=<id> next to each record it manages, but if two owner TXT records ever exist for the same name, the registry silently keeps only one in memory, and the tool can then reconcile a record it does not actually own. The fix is to give every managed record an explicit, checkable owner tag and refuse to write when that tag is missing or belongs to someone else. Full code, tests, and the decision logic are below.
The problem in plain words
Automated DNS tools work by comparing two lists: what should exist (the desired state, usually a Kubernetes Ingress or a Terraform file) and what actually exists in the zone right now. Wherever those two lists disagree, the tool writes a change to make the zone match its desired state. That is the whole job.
The trouble starts when the tool assumes every record with a matching name and type is "its" record, without checking who actually put it there. A record a person added by hand years ago, a record another team's separate ExternalDNS instance is managing, or a record left by a tool that has since been decommissioned, can all look, on the surface, like a record the current reconciler should manage. If there is no marker saying otherwise, the reconciler treats it as fair game and mutates or deletes it during its normal reconcile pass.
Why it happens
ExternalDNS ships a TXT registry mechanism precisely to solve this: a companion TXT record next to each managed name that stores heritage=external-dns,external-dns/owner=<id>, so the tool can tell its own records apart from anyone else's. Even with that in place, the ownership check can still fail in practice:
- Two TXT ownership records end up coexisting for the same name and type, for example after a failed apply, a migration between clusters, or a race between two runs. ExternalDNS parses these into an in-memory map keyed by name, and the second one read silently overwrites the first, so the tool can end up trusting the wrong owner. This is the exact scenario in kubernetes-sigs/external-dns issue #5457.
- A record was created outside ExternalDNS entirely, by hand in the provider's dashboard or by a completely different tool, so it has no TXT ownership record at all, and older or misconfigured versions did not always treat "no marker" as "leave it alone."
- The registry read order is not guaranteed, so which owner "wins" the in-memory map can differ from one reconcile run to the next, meaning the tool's idea of what it owns can flip without any code change on either side.
- A custom or hand-rolled sync script skips the ownership check altogether and just diffs intended values against live values, since that is simpler to write than checking a marker first.
The result is DNS drift that looks random: a record flips back and forth, or disappears, depending on which owner's TXT record the registry happened to read last, not because anyone told the system to reassign it.
A reconciler should only ever be allowed to change a record it can prove it owns. "Matches by name and type" is not proof of ownership, it is just a coincidence of naming. The only safe proof is an explicit marker, a TXT ownership record or a comment/tag on the record itself, that the reconciler checks before every write and refuses to proceed without.
The fix, as a flow
Before any script or controller changes a record, it should look up what is live, look up the ownership marker for that name, and only proceed if the marker matches its own owner id. If the marker is missing or belongs to someone else, the safe move is to skip the write and log a conflict, not to write anyway and hope for the best.
How to fix it
List what the reconciler currently believes it owns vs what is live
Pull the live records for the name in question from Cloudflare and read the comment and tags fields. For an ExternalDNS-managed zone, check the sibling TXT record instead, since that is where the ownership marker actually lives.
curl -s -H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?name=app.example.com" \
| jq '.result[] | {id, type, content, comment, tags}'
Check for a matching TXT ownership record
Look up the TXT record ExternalDNS writes next to the managed name, using whatever --txt-prefix you configured. A healthy zone has exactly one TXT value for that name. Two or more TXT records with different owner= values is the exact collision behind kubernetes-sigs/external-dns issue #5457.
dig +short TXT app.example.com @1.1.1.1
# or, with a registry prefix such as extdns-:
dig +short TXT extdns-app.example.com @1.1.1.1
# a healthy result is exactly one line, like:
# "heritage=external-dns,external-dns/owner=team-a"
# a bad result is two or more lines with different owner= values, for example:
# "heritage=external-dns,external-dns/owner=team-a"
# "heritage=external-dns,external-dns/owner=team-b"
Confirm what is actually live vs. what was intended
Compare the current live value to your source of truth, the Terraform state or the Kubernetes Ingress spec. A live value that does not match intent, with no ownership marker to explain who is allowed to change it, is unowned-record drift.
dig +short A app.example.com @1.1.1.1
# compare against your source of truth, e.g.:
terraform state show cloudflare_dns_record.app_example_com
Look for unexpected plan diffs in Terraform
In Terraform-managed zones, an unexpected in-place update or destroy of a cloudflare_dns_record resource nobody remembers defining is the reconciliation equivalent of the same bug: the provider is about to write over a record based on a stale or mismatched view of what it owns.
terraform plan -target=cloudflare_dns_record.app_example_com
# an unexpected "~ update in-place" or "- destroy" here,
# on a resource nobody remembers defining, is the same bug
Run ExternalDNS with an explicit owner id and upsert-only
Give the reconciler its own owner id and TXT prefix, and keep the policy to upsert-only until you have verified ownership across the zone. Upsert-only means it will create or update records, but never delete one, which limits the blast radius while you sort out ownership.
external-dns \
--registry=txt \
--txt-owner-id=team-a \
--txt-prefix=extdns- \
--policy=upsert-only \
--provider=cloudflare
Delete the stale duplicate TXT ownership record
If two conflicting TXT owner records exist for the same name, the reconciler cannot fix that on its own. Manually delete the stale one through the DNS provider so only one owner record remains before letting the controller run again.
extdns-app.example.com. 300 IN TXT "heritage=external-dns,external-dns/owner=team-a"
; the stale duplicate below must be removed by hand at the provider:
; extdns-app.example.com. 300 IN TXT "heritage=external-dns,external-dns/owner=team-b"
Guard a custom script with a comment or tag check
For a hand-rolled sync script that is not ExternalDNS, before writing any change, do a GET against the Cloudflare API for that record and require a comment such as managed-by:pipeline-x (or a matching tag) to already be present. Only when it matches do you proceed to PATCH or POST. Otherwise skip and log a conflict.
; live record, as returned by GET /zones/{zone}/dns_records
app.example.com. 300 IN A 203.0.113.10 ; comment: managed-by:pipeline-x
How to check it worked
Confirm there is exactly one ownership marker per managed name, that the live value matches your intended state, and that repeated reconciler runs do not flip it back and forth.
# exactly one TXT ownership string with the expected owner
dig +short TXT extdns-app.example.com @1.1.1.1
# the A record should match your source of truth
dig +short A app.example.com @1.1.1.1
# run it again 60+ seconds later, confirm it has not flip-flopped
dig +short A app.example.com @1.1.1.1
# every record the automation may touch should carry the ownership comment
curl -s -H "Authorization: Bearer $CF_TOKEN" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?name=app.example.com" \
| jq '.result[].comment'
# re-run in dry-run/plan mode and confirm zero unexpected diffs
terraform plan
# or: external-dns --dry-run
A good result looks like this: dig +short TXT returns exactly one string with the owner you expect, dig +short A matches your intended value on both runs with no flip-flopping, the comment field shows your ownership tag on every record the automation should touch and nothing on records it should leave alone, and a dry-run pass shows zero diffs against records belonging to other owners.
The full code
Here is the complete checker in one file for each language. It fetches the live record and its ownership marker from Cloudflare, decides with a pure function whether a write is safe, and only proceeds to create or update when the marker matches. Everything else is logged as a skipped conflict. It stays in dry run by default so it reports before it writes.
View this code on GitHub Full runnable folder with tests in the dns-fixes repo.
"""Detect a DNS record a reconciler is about to write without proof of
ownership, and repair it via Cloudflare only when the ownership marker
matches. Safe to run on a schedule. Stays in dry run until DRY_RUN=false.
"""
import os
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("check_record_ownership")
def decide_action(intended: dict, live: dict | None, owner_id: str) -> str:
"""
intended: {"name": str, "type": str, "content": str, "owner": str}
live: {"name": str, "type": str, "content": str, "comment": str|None} or None if absent
owner_id: this reconciler's own owner tag, e.g. "team-a"
Returns one of: "create", "update", "skip_conflict", "noop"
Pure decision logic, no I/O: given the intended record, the current live record
(or None), and this instance's owner id, decide whether it is safe to write.
"""
if live is None:
return "create"
live_comment = live.get("comment") or ""
live_owner = live_comment.split("managed-by:")[-1].strip() if "managed-by:" in live_comment else None
if live_owner is None:
return "skip_conflict"
if live_owner != owner_id:
return "skip_conflict"
if live.get("content") == intended.get("content"):
return "noop"
return "update"
def run():
# Imported lazily so the pure function above can be tested with no
# network libraries installed at all.
import requests
zone_id = os.environ["CLOUDFLARE_ZONE_ID"]
api_token = os.environ["CLOUDFLARE_API_TOKEN"]
owner_id = os.environ.get("DNS_OWNER_ID", "team-a")
dry_run = os.environ.get("DRY_RUN", "true").lower() == "true"
# In a real run, "desired" would come from Terraform state or a
# Kubernetes Ingress spec. Here it is one example record.
desired = [
{"name": "app.example.com", "type": "A", "content": "203.0.113.10", "owner": owner_id},
]
headers = {"Authorization": f"Bearer {api_token}", "Content-Type": "application/json"}
base = f"https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records"
for intended in desired:
resp = requests.get(base, headers=headers, params={"name": intended["name"], "type": intended["type"]}, timeout=30)
resp.raise_for_status()
results = resp.json().get("result", [])
live = results[0] if results else None
action = decide_action(intended, live, owner_id)
log.info("Record %s (%s): action=%s", intended["name"], intended["type"], action)
if action == "noop":
log.info("Already correct and owned. Nothing to do.")
continue
if action == "skip_conflict":
log.warning(
"Skipping %s: live record has no matching owner marker for '%s'. "
"Refusing to overwrite a record this reconciler does not own.",
intended["name"], owner_id,
)
continue
comment = f"managed-by:{owner_id}"
if action == "create":
log.info("%s create %s -> %s", "Would" if dry_run else "Will", intended["name"], intended["content"])
if not dry_run:
requests.post(
base, headers=headers,
json={"type": intended["type"], "name": intended["name"], "content": intended["content"], "ttl": 300, "comment": comment},
timeout=30,
).raise_for_status()
elif action == "update":
log.info("%s update %s -> %s", "Would" if dry_run else "Will", intended["name"], intended["content"])
if not dry_run:
requests.patch(
f"{base}/{live['id']}", headers=headers,
json={"content": intended["content"], "comment": comment},
timeout=30,
).raise_for_status()
log.info("Done.")
if __name__ == "__main__":
run()
/**
* Detect a DNS record a reconciler is about to write without proof of
* ownership, and repair it via Cloudflare only when the ownership marker
* matches. Safe to run on a schedule. Stays in dry run until DRY_RUN=false.
*/
import { pathToFileURL } from "node:url";
/**
* intended: { name: string, type: string, content: string, owner: string }
* live: { name: string, type: string, content: string, comment: string|null } or null if absent
* ownerId: this reconciler's own owner tag, e.g. "team-a"
* Returns one of: "create", "update", "skip_conflict", "noop"
* Pure decision logic, no I/O: given the intended record, the current live record
* (or null), and this instance's owner id, decide whether it is safe to write.
*/
export function decideAction(intended, live, ownerId) {
if (live === null || live === undefined) {
return "create";
}
const liveComment = live.comment || "";
const liveOwner = liveComment.includes("managed-by:")
? liveComment.split("managed-by:").pop().trim()
: null;
if (liveOwner === null) {
return "skip_conflict";
}
if (liveOwner !== ownerId) {
return "skip_conflict";
}
if (live.content === intended.content) {
return "noop";
}
return "update";
}
export async function run() {
const zoneId = process.env.CLOUDFLARE_ZONE_ID;
const apiToken = process.env.CLOUDFLARE_API_TOKEN;
const ownerId = process.env.DNS_OWNER_ID || "team-a";
const dryRun = (process.env.DRY_RUN || "true").toLowerCase() === "true";
// In a real run, "desired" would come from Terraform state or a
// Kubernetes Ingress spec. Here it is one example record.
const desired = [
{ name: "app.example.com", type: "A", content: "203.0.113.10", owner: ownerId },
];
const headers = { Authorization: `Bearer ${apiToken}`, "Content-Type": "application/json" };
const base = `https://api.cloudflare.com/client/v4/zones/${zoneId}/dns_records`;
for (const intended of desired) {
const listUrl = `${base}?name=${encodeURIComponent(intended.name)}&type=${intended.type}`;
const listRes = await fetch(listUrl, { headers });
if (!listRes.ok) throw new Error(`Cloudflare list failed: ${listRes.status}`);
const listBody = await listRes.json();
const results = listBody.result || [];
const live = results.length > 0 ? results[0] : null;
const action = decideAction(intended, live, ownerId);
console.log(`Record ${intended.name} (${intended.type}): action=${action}`);
if (action === "noop") {
console.log("Already correct and owned. Nothing to do.");
continue;
}
if (action === "skip_conflict") {
console.warn(
`Skipping ${intended.name}: live record has no matching owner marker for '${ownerId}'. ` +
`Refusing to overwrite a record this reconciler does not own.`
);
continue;
}
const comment = `managed-by:${ownerId}`;
if (action === "create") {
console.log(`${dryRun ? "Would" : "Will"} create ${intended.name} -> ${intended.content}`);
if (!dryRun) {
const res = await fetch(base, {
method: "POST",
headers,
body: JSON.stringify({ type: intended.type, name: intended.name, content: intended.content, ttl: 300, comment }),
});
if (!res.ok) throw new Error(`Cloudflare create failed: ${res.status}`);
}
} else if (action === "update") {
console.log(`${dryRun ? "Would" : "Will"} update ${intended.name} -> ${intended.content}`);
if (!dryRun) {
const res = await fetch(`${base}/${live.id}`, {
method: "PATCH",
headers,
body: JSON.stringify({ content: intended.content, comment }),
});
if (!res.ok) throw new Error(`Cloudflare update failed: ${res.status}`);
}
}
}
console.log("Done.");
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((e) => { console.error(e); process.exit(1); });
}
Add a test
The part worth testing is the decision function itself, since it decides whether a real record gets overwritten. It takes plain objects in and returns a plain string, so the test needs no network and no Cloudflare account.
from check_record_ownership import decide_action
def intended(**over):
base = {"name": "app.example.com", "type": "A", "content": "203.0.113.10", "owner": "team-a"}
base.update(over)
return base
def test_create_when_no_live_record():
assert decide_action(intended(), None, "team-a") == "create"
def test_noop_when_owned_and_matching():
live = {"name": "app.example.com", "type": "A", "content": "203.0.113.10", "comment": "managed-by:team-a"}
assert decide_action(intended(), live, "team-a") == "noop"
def test_update_when_owned_and_content_differs():
live = {"name": "app.example.com", "type": "A", "content": "198.51.100.5", "comment": "managed-by:team-a"}
assert decide_action(intended(), live, "team-a") == "update"
def test_skip_conflict_when_owner_differs():
live = {"name": "app.example.com", "type": "A", "content": "198.51.100.5", "comment": "managed-by:team-b"}
assert decide_action(intended(), live, "team-a") == "skip_conflict"
def test_skip_conflict_when_no_ownership_marker():
live = {"name": "app.example.com", "type": "A", "content": "198.51.100.5", "comment": None}
assert decide_action(intended(), live, "team-a") == "skip_conflict"
import { test } from "node:test";
import assert from "node:assert/strict";
import { decideAction } from "./check-record-ownership.js";
const intended = (over = {}) => ({
name: "app.example.com", type: "A", content: "203.0.113.10", owner: "team-a", ...over,
});
test("create when no live record", () => {
assert.equal(decideAction(intended(), null, "team-a"), "create");
});
test("noop when owned and matching", () => {
const live = { name: "app.example.com", type: "A", content: "203.0.113.10", comment: "managed-by:team-a" };
assert.equal(decideAction(intended(), live, "team-a"), "noop");
});
test("update when owned and content differs", () => {
const live = { name: "app.example.com", type: "A", content: "198.51.100.5", comment: "managed-by:team-a" };
assert.equal(decideAction(intended(), live, "team-a"), "update");
});
test("skip_conflict when owner differs", () => {
const live = { name: "app.example.com", type: "A", content: "198.51.100.5", comment: "managed-by:team-b" };
assert.equal(decideAction(intended(), live, "team-a"), "skip_conflict");
});
test("skip_conflict when no ownership marker", () => {
const live = { name: "app.example.com", type: "A", content: "198.51.100.5", comment: null };
assert.equal(decideAction(intended(), live, "team-a"), "skip_conflict");
});
Case studies
Two clusters, one name, one TXT record left behind
A platform team migrated ExternalDNS from an old Kubernetes cluster to a new one, using the same owner id during the cutover window to avoid downtime. The old cluster's TXT ownership record was never cleaned up, so for a few days two TXT records with different owner values sat on the same name.
The new cluster's reconciler read the wrong owner on one reconcile pass and deleted a CNAME the old cluster still needed. Deleting the stale TXT record and switching to distinct owner ids per cluster during migrations stopped the collision for good.
A support subdomain a person added years ago
A support engineer added an A record for a status page by hand, directly in the Cloudflare dashboard, long before the team adopted ExternalDNS. The record had no ownership comment and no TXT marker, since it predated the automation entirely.
When the team's Terraform provider imported the zone, its plan quietly proposed a destroy on that record because it matched the name pattern used elsewhere. Adding a required ownership comment check before any write caught the missing marker and skipped the record instead.
Once every write is gated on a matching ownership marker, a reconciler stops being able to touch records it does not own, even by accident. Records with no marker, or a marker belonging to someone else, are reported as conflicts and left alone. The only records that ever change are the ones the tool can prove are its own.
FAQ
Why did my automation delete or change a DNS record I did not touch?
A reconciler like ExternalDNS or a Terraform provider manages a zone by comparing a desired state to the live records and writing whatever is different. If it does not first check an ownership marker, it can mistake a record created by a human, another team, or a different tool for one of its own and change or delete it.
What is the TXT registry in ExternalDNS and why does it fail?
ExternalDNS can write a companion TXT record next to each managed record holding heritage=external-dns,external-dns/owner=<id>. If two TXT records with different owner values ever exist for the same name, ExternalDNS parses them into a map that silently keeps only one, so the tool can misjudge what it owns and reconcile a record it should have left alone.
How do I stop a reconciler from overwriting records it should not touch?
Run it with an explicit owner id and a TXT registry, keep the policy to upsert-only until ownership is verified, and make any custom script check a comment or tag field on the live record before writing. If it does not match the expected owner, skip the write and log a conflict instead of overwriting.
Related field notes
Citations
On the problem:
- kubernetes-sigs/external-dns Issue #5457: In TXTRegistry, silently overwritten ownership data can lead to unpredictable behavior when multiple TXT records exist. github.com/kubernetes-sigs/external-dns/issues/5457
- external-dns TXT Registry documentation. github.com/kubernetes-sigs/external-dns/blob/master/docs/registry/txt.md
- external-dns Issue #3715: external-dns not ignoring pre-existing DNS records it does not have ownership of. github.com/kubernetes-sigs/external-dns/issues/3715
On the solution:
- Cloudflare API: List DNS Records. developers.cloudflare.com/api/resources/dns/subresources/records/methods/list
- Terraform Registry: cloudflare_dns_record resource. registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/dns_record
- external-dns FAQ, registry, txt-owner-id, and policy flags. kubernetes-sigs.github.io/external-dns/v0.12.2/faq
Stuck on a tricky one?
If you have a DNS automation, reconciler, or ownership drift problem you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this stop a record from being overwritten?
If this saved you a scary reconcile diff or a record that kept flipping back and forth, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi