Reconciler Provider API / Reconciliation

DNS-managed record overwritten without an ownership check

A record that a person created by hand, or that another team's automation owns, gets quietly changed or deleted by a tool like ExternalDNS or a Terraform provider. Nobody asked for the change. Nobody reassigned the record on purpose. The reconciler just diffed its desired state against the live zone and wrote over whatever did not match, because it never checked who was actually allowed to touch that name first.

Python and Node.js Cloudflare DNS API Safe by default (dry run)
Server cabinets
Photo by Eric Stoynov on Unsplash
The short answer

Reconcilers like ExternalDNS and Terraform manage a zone by comparing a desired state to what is live, then writing whatever is different. If the tool never checks an ownership marker before writing, it can overwrite or delete a record created by a human, a different team's automation, or a different tool entirely. ExternalDNS's own TXT registry is supposed to prevent this by storing heritage=external-dns,external-dns/owner=<id> next to each record it manages, but if two owner TXT records ever exist for the same name, the registry silently keeps only one in memory, and the tool can then reconcile a record it does not actually own. The fix is to give every managed record an explicit, checkable owner tag and refuse to write when that tag is missing or belongs to someone else. Full code, tests, and the decision logic are below.

The problem in plain words

Automated DNS tools work by comparing two lists: what should exist (the desired state, usually a Kubernetes Ingress or a Terraform file) and what actually exists in the zone right now. Wherever those two lists disagree, the tool writes a change to make the zone match its desired state. That is the whole job.

The trouble starts when the tool assumes every record with a matching name and type is "its" record, without checking who actually put it there. A record a person added by hand years ago, a record another team's separate ExternalDNS instance is managing, or a record left by a tool that has since been decommissioned, can all look, on the surface, like a record the current reconciler should manage. If there is no marker saying otherwise, the reconciler treats it as fair game and mutates or deletes it during its normal reconcile pass.

Reconciler runs diffs desired vs live Finds matching name no owner check done assumed to be mine Record was someone else's Overwritten or deleted no reassignment was requested, it happened as a side effect of a routine reconcile pass
The reconciler is only doing its normal job, comparing desired state to live state. The bug is that it never asked who else might already own the name it is about to change.

Why it happens

ExternalDNS ships a TXT registry mechanism precisely to solve this: a companion TXT record next to each managed name that stores heritage=external-dns,external-dns/owner=<id>, so the tool can tell its own records apart from anyone else's. Even with that in place, the ownership check can still fail in practice:

The result is DNS drift that looks random: a record flips back and forth, or disappears, depending on which owner's TXT record the registry happened to read last, not because anyone told the system to reassign it.

The key insight

A reconciler should only ever be allowed to change a record it can prove it owns. "Matches by name and type" is not proof of ownership, it is just a coincidence of naming. The only safe proof is an explicit marker, a TXT ownership record or a comment/tag on the record itself, that the reconciler checks before every write and refuses to proceed without.

The fix, as a flow

Before any script or controller changes a record, it should look up what is live, look up the ownership marker for that name, and only proceed if the marker matches its own owner id. If the marker is missing or belongs to someone else, the safe move is to skip the write and log a conflict, not to write anyway and hope for the best.

Intended record from desired state Fetch live record + its owner marker Compare owner id to this reconciler's own Marker matches? yes no, skip conflict Create or update only the owned record
Every write is gated on an ownership check. A missing or mismatched marker is logged and skipped, never overwritten.

How to fix it

1

List what the reconciler currently believes it owns vs what is live

Pull the live records for the name in question from Cloudflare and read the comment and tags fields. For an ExternalDNS-managed zone, check the sibling TXT record instead, since that is where the ownership marker actually lives.

Terminal
list-live-records.sh
curl -s -H "Authorization: Bearer $CF_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?name=app.example.com" \
  | jq '.result[] | {id, type, content, comment, tags}'
2

Check for a matching TXT ownership record

Look up the TXT record ExternalDNS writes next to the managed name, using whatever --txt-prefix you configured. A healthy zone has exactly one TXT value for that name. Two or more TXT records with different owner= values is the exact collision behind kubernetes-sigs/external-dns issue #5457.

Terminal
check-txt-ownership.sh
dig +short TXT app.example.com @1.1.1.1

# or, with a registry prefix such as extdns-:
dig +short TXT extdns-app.example.com @1.1.1.1

# a healthy result is exactly one line, like:
# "heritage=external-dns,external-dns/owner=team-a"

# a bad result is two or more lines with different owner= values, for example:
# "heritage=external-dns,external-dns/owner=team-a"
# "heritage=external-dns,external-dns/owner=team-b"
3

Confirm what is actually live vs. what was intended

Compare the current live value to your source of truth, the Terraform state or the Kubernetes Ingress spec. A live value that does not match intent, with no ownership marker to explain who is allowed to change it, is unowned-record drift.

Terminal
check-live-vs-intended.sh
dig +short A app.example.com @1.1.1.1

# compare against your source of truth, e.g.:
terraform state show cloudflare_dns_record.app_example_com
4

Look for unexpected plan diffs in Terraform

In Terraform-managed zones, an unexpected in-place update or destroy of a cloudflare_dns_record resource nobody remembers defining is the reconciliation equivalent of the same bug: the provider is about to write over a record based on a stale or mismatched view of what it owns.

Terminal
terraform-plan-check.sh
terraform plan -target=cloudflare_dns_record.app_example_com

# an unexpected "~ update in-place" or "- destroy" here,
# on a resource nobody remembers defining, is the same bug
5

Run ExternalDNS with an explicit owner id and upsert-only

Give the reconciler its own owner id and TXT prefix, and keep the policy to upsert-only until you have verified ownership across the zone. Upsert-only means it will create or update records, but never delete one, which limits the blast radius while you sort out ownership.

Terminal
external-dns-flags.sh
external-dns \
  --registry=txt \
  --txt-owner-id=team-a \
  --txt-prefix=extdns- \
  --policy=upsert-only \
  --provider=cloudflare
6

Delete the stale duplicate TXT ownership record

If two conflicting TXT owner records exist for the same name, the reconciler cannot fix that on its own. Manually delete the stale one through the DNS provider so only one owner record remains before letting the controller run again.

DNS record
zone record (after cleanup)
extdns-app.example.com.  300  IN  TXT  "heritage=external-dns,external-dns/owner=team-a"

; the stale duplicate below must be removed by hand at the provider:
; extdns-app.example.com.  300  IN  TXT  "heritage=external-dns,external-dns/owner=team-b"
7

Guard a custom script with a comment or tag check

For a hand-rolled sync script that is not ExternalDNS, before writing any change, do a GET against the Cloudflare API for that record and require a comment such as managed-by:pipeline-x (or a matching tag) to already be present. Only when it matches do you proceed to PATCH or POST. Otherwise skip and log a conflict.

DNS record
expected record (with ownership comment)
; live record, as returned by GET /zones/{zone}/dns_records
app.example.com.  300  IN  A  203.0.113.10   ; comment: managed-by:pipeline-x

How to check it worked

Confirm there is exactly one ownership marker per managed name, that the live value matches your intended state, and that repeated reconciler runs do not flip it back and forth.

Terminal
verify.sh
# exactly one TXT ownership string with the expected owner
dig +short TXT extdns-app.example.com @1.1.1.1

# the A record should match your source of truth
dig +short A app.example.com @1.1.1.1

# run it again 60+ seconds later, confirm it has not flip-flopped
dig +short A app.example.com @1.1.1.1

# every record the automation may touch should carry the ownership comment
curl -s -H "Authorization: Bearer $CF_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?name=app.example.com" \
  | jq '.result[].comment'

# re-run in dry-run/plan mode and confirm zero unexpected diffs
terraform plan
# or: external-dns --dry-run

A good result looks like this: dig +short TXT returns exactly one string with the owner you expect, dig +short A matches your intended value on both runs with no flip-flopping, the comment field shows your ownership tag on every record the automation should touch and nothing on records it should leave alone, and a dry-run pass shows zero diffs against records belonging to other owners.

The full code

Here is the complete checker in one file for each language. It fetches the live record and its ownership marker from Cloudflare, decides with a pure function whether a write is safe, and only proceeds to create or update when the marker matches. Everything else is logged as a skipped conflict. It stays in dry run by default so it reports before it writes.

View this code on GitHub Full runnable folder with tests in the dns-fixes repo.

check_record_ownership.py
"""Detect a DNS record a reconciler is about to write without proof of
ownership, and repair it via Cloudflare only when the ownership marker
matches. Safe to run on a schedule. Stays in dry run until DRY_RUN=false.
"""
import os
import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("check_record_ownership")


def decide_action(intended: dict, live: dict | None, owner_id: str) -> str:
    """
    intended: {"name": str, "type": str, "content": str, "owner": str}
    live: {"name": str, "type": str, "content": str, "comment": str|None} or None if absent
    owner_id: this reconciler's own owner tag, e.g. "team-a"
    Returns one of: "create", "update", "skip_conflict", "noop"
    Pure decision logic, no I/O: given the intended record, the current live record
    (or None), and this instance's owner id, decide whether it is safe to write.
    """
    if live is None:
        return "create"

    live_comment = live.get("comment") or ""
    live_owner = live_comment.split("managed-by:")[-1].strip() if "managed-by:" in live_comment else None

    if live_owner is None:
        return "skip_conflict"
    if live_owner != owner_id:
        return "skip_conflict"
    if live.get("content") == intended.get("content"):
        return "noop"
    return "update"


def run():
    # Imported lazily so the pure function above can be tested with no
    # network libraries installed at all.
    import requests

    zone_id = os.environ["CLOUDFLARE_ZONE_ID"]
    api_token = os.environ["CLOUDFLARE_API_TOKEN"]
    owner_id = os.environ.get("DNS_OWNER_ID", "team-a")
    dry_run = os.environ.get("DRY_RUN", "true").lower() == "true"

    # In a real run, "desired" would come from Terraform state or a
    # Kubernetes Ingress spec. Here it is one example record.
    desired = [
        {"name": "app.example.com", "type": "A", "content": "203.0.113.10", "owner": owner_id},
    ]

    headers = {"Authorization": f"Bearer {api_token}", "Content-Type": "application/json"}
    base = f"https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records"

    for intended in desired:
        resp = requests.get(base, headers=headers, params={"name": intended["name"], "type": intended["type"]}, timeout=30)
        resp.raise_for_status()
        results = resp.json().get("result", [])
        live = results[0] if results else None

        action = decide_action(intended, live, owner_id)
        log.info("Record %s (%s): action=%s", intended["name"], intended["type"], action)

        if action == "noop":
            log.info("Already correct and owned. Nothing to do.")
            continue

        if action == "skip_conflict":
            log.warning(
                "Skipping %s: live record has no matching owner marker for '%s'. "
                "Refusing to overwrite a record this reconciler does not own.",
                intended["name"], owner_id,
            )
            continue

        comment = f"managed-by:{owner_id}"
        if action == "create":
            log.info("%s create %s -> %s", "Would" if dry_run else "Will", intended["name"], intended["content"])
            if not dry_run:
                requests.post(
                    base, headers=headers,
                    json={"type": intended["type"], "name": intended["name"], "content": intended["content"], "ttl": 300, "comment": comment},
                    timeout=30,
                ).raise_for_status()
        elif action == "update":
            log.info("%s update %s -> %s", "Would" if dry_run else "Will", intended["name"], intended["content"])
            if not dry_run:
                requests.patch(
                    f"{base}/{live['id']}", headers=headers,
                    json={"content": intended["content"], "comment": comment},
                    timeout=30,
                ).raise_for_status()

    log.info("Done.")


if __name__ == "__main__":
    run()
check-record-ownership.js
/**
 * Detect a DNS record a reconciler is about to write without proof of
 * ownership, and repair it via Cloudflare only when the ownership marker
 * matches. Safe to run on a schedule. Stays in dry run until DRY_RUN=false.
 */
import { pathToFileURL } from "node:url";

/**
 * intended: { name: string, type: string, content: string, owner: string }
 * live: { name: string, type: string, content: string, comment: string|null } or null if absent
 * ownerId: this reconciler's own owner tag, e.g. "team-a"
 * Returns one of: "create", "update", "skip_conflict", "noop"
 * Pure decision logic, no I/O: given the intended record, the current live record
 * (or null), and this instance's owner id, decide whether it is safe to write.
 */
export function decideAction(intended, live, ownerId) {
  if (live === null || live === undefined) {
    return "create";
  }

  const liveComment = live.comment || "";
  const liveOwner = liveComment.includes("managed-by:")
    ? liveComment.split("managed-by:").pop().trim()
    : null;

  if (liveOwner === null) {
    return "skip_conflict";
  }
  if (liveOwner !== ownerId) {
    return "skip_conflict";
  }
  if (live.content === intended.content) {
    return "noop";
  }
  return "update";
}

export async function run() {
  const zoneId = process.env.CLOUDFLARE_ZONE_ID;
  const apiToken = process.env.CLOUDFLARE_API_TOKEN;
  const ownerId = process.env.DNS_OWNER_ID || "team-a";
  const dryRun = (process.env.DRY_RUN || "true").toLowerCase() === "true";

  // In a real run, "desired" would come from Terraform state or a
  // Kubernetes Ingress spec. Here it is one example record.
  const desired = [
    { name: "app.example.com", type: "A", content: "203.0.113.10", owner: ownerId },
  ];

  const headers = { Authorization: `Bearer ${apiToken}`, "Content-Type": "application/json" };
  const base = `https://api.cloudflare.com/client/v4/zones/${zoneId}/dns_records`;

  for (const intended of desired) {
    const listUrl = `${base}?name=${encodeURIComponent(intended.name)}&type=${intended.type}`;
    const listRes = await fetch(listUrl, { headers });
    if (!listRes.ok) throw new Error(`Cloudflare list failed: ${listRes.status}`);
    const listBody = await listRes.json();
    const results = listBody.result || [];
    const live = results.length > 0 ? results[0] : null;

    const action = decideAction(intended, live, ownerId);
    console.log(`Record ${intended.name} (${intended.type}): action=${action}`);

    if (action === "noop") {
      console.log("Already correct and owned. Nothing to do.");
      continue;
    }

    if (action === "skip_conflict") {
      console.warn(
        `Skipping ${intended.name}: live record has no matching owner marker for '${ownerId}'. ` +
        `Refusing to overwrite a record this reconciler does not own.`
      );
      continue;
    }

    const comment = `managed-by:${ownerId}`;
    if (action === "create") {
      console.log(`${dryRun ? "Would" : "Will"} create ${intended.name} -> ${intended.content}`);
      if (!dryRun) {
        const res = await fetch(base, {
          method: "POST",
          headers,
          body: JSON.stringify({ type: intended.type, name: intended.name, content: intended.content, ttl: 300, comment }),
        });
        if (!res.ok) throw new Error(`Cloudflare create failed: ${res.status}`);
      }
    } else if (action === "update") {
      console.log(`${dryRun ? "Would" : "Will"} update ${intended.name} -> ${intended.content}`);
      if (!dryRun) {
        const res = await fetch(`${base}/${live.id}`, {
          method: "PATCH",
          headers,
          body: JSON.stringify({ content: intended.content, comment }),
        });
        if (!res.ok) throw new Error(`Cloudflare update failed: ${res.status}`);
      }
    }
  }

  console.log("Done.");
}

if (import.meta.url === pathToFileURL(process.argv[1]).href) {
  run().catch((e) => { console.error(e); process.exit(1); });
}

Add a test

The part worth testing is the decision function itself, since it decides whether a real record gets overwritten. It takes plain objects in and returns a plain string, so the test needs no network and no Cloudflare account.

test_ownership_decision.py
from check_record_ownership import decide_action


def intended(**over):
    base = {"name": "app.example.com", "type": "A", "content": "203.0.113.10", "owner": "team-a"}
    base.update(over)
    return base


def test_create_when_no_live_record():
    assert decide_action(intended(), None, "team-a") == "create"


def test_noop_when_owned_and_matching():
    live = {"name": "app.example.com", "type": "A", "content": "203.0.113.10", "comment": "managed-by:team-a"}
    assert decide_action(intended(), live, "team-a") == "noop"


def test_update_when_owned_and_content_differs():
    live = {"name": "app.example.com", "type": "A", "content": "198.51.100.5", "comment": "managed-by:team-a"}
    assert decide_action(intended(), live, "team-a") == "update"


def test_skip_conflict_when_owner_differs():
    live = {"name": "app.example.com", "type": "A", "content": "198.51.100.5", "comment": "managed-by:team-b"}
    assert decide_action(intended(), live, "team-a") == "skip_conflict"


def test_skip_conflict_when_no_ownership_marker():
    live = {"name": "app.example.com", "type": "A", "content": "198.51.100.5", "comment": None}
    assert decide_action(intended(), live, "team-a") == "skip_conflict"
ownership-decision.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { decideAction } from "./check-record-ownership.js";

const intended = (over = {}) => ({
  name: "app.example.com", type: "A", content: "203.0.113.10", owner: "team-a", ...over,
});

test("create when no live record", () => {
  assert.equal(decideAction(intended(), null, "team-a"), "create");
});

test("noop when owned and matching", () => {
  const live = { name: "app.example.com", type: "A", content: "203.0.113.10", comment: "managed-by:team-a" };
  assert.equal(decideAction(intended(), live, "team-a"), "noop");
});

test("update when owned and content differs", () => {
  const live = { name: "app.example.com", type: "A", content: "198.51.100.5", comment: "managed-by:team-a" };
  assert.equal(decideAction(intended(), live, "team-a"), "update");
});

test("skip_conflict when owner differs", () => {
  const live = { name: "app.example.com", type: "A", content: "198.51.100.5", comment: "managed-by:team-b" };
  assert.equal(decideAction(intended(), live, "team-a"), "skip_conflict");
});

test("skip_conflict when no ownership marker", () => {
  const live = { name: "app.example.com", type: "A", content: "198.51.100.5", comment: null };
  assert.equal(decideAction(intended(), live, "team-a"), "skip_conflict");
});

Case studies

Migration collision

Two clusters, one name, one TXT record left behind

A platform team migrated ExternalDNS from an old Kubernetes cluster to a new one, using the same owner id during the cutover window to avoid downtime. The old cluster's TXT ownership record was never cleaned up, so for a few days two TXT records with different owner values sat on the same name.

The new cluster's reconciler read the wrong owner on one reconcile pass and deleted a CNAME the old cluster still needed. Deleting the stale TXT record and switching to distinct owner ids per cluster during migrations stopped the collision for good.

Hand-added record

A support subdomain a person added years ago

A support engineer added an A record for a status page by hand, directly in the Cloudflare dashboard, long before the team adopted ExternalDNS. The record had no ownership comment and no TXT marker, since it predated the automation entirely.

When the team's Terraform provider imported the zone, its plan quietly proposed a destroy on that record because it matched the name pattern used elsewhere. Adding a required ownership comment check before any write caught the missing marker and skipped the record instead.

What good looks like

Once every write is gated on a matching ownership marker, a reconciler stops being able to touch records it does not own, even by accident. Records with no marker, or a marker belonging to someone else, are reported as conflicts and left alone. The only records that ever change are the ones the tool can prove are its own.

FAQ

Why did my automation delete or change a DNS record I did not touch?

A reconciler like ExternalDNS or a Terraform provider manages a zone by comparing a desired state to the live records and writing whatever is different. If it does not first check an ownership marker, it can mistake a record created by a human, another team, or a different tool for one of its own and change or delete it.

What is the TXT registry in ExternalDNS and why does it fail?

ExternalDNS can write a companion TXT record next to each managed record holding heritage=external-dns,external-dns/owner=<id>. If two TXT records with different owner values ever exist for the same name, ExternalDNS parses them into a map that silently keeps only one, so the tool can misjudge what it owns and reconcile a record it should have left alone.

How do I stop a reconciler from overwriting records it should not touch?

Run it with an explicit owner id and a TXT registry, keep the policy to upsert-only until ownership is verified, and make any custom script check a comment or tag field on the live record before writing. If it does not match the expected owner, skip the write and log a conflict instead of overwriting.

Related field notes

Citations

On the problem:

  1. kubernetes-sigs/external-dns Issue #5457: In TXTRegistry, silently overwritten ownership data can lead to unpredictable behavior when multiple TXT records exist. github.com/kubernetes-sigs/external-dns/issues/5457
  2. external-dns TXT Registry documentation. github.com/kubernetes-sigs/external-dns/blob/master/docs/registry/txt.md
  3. external-dns Issue #3715: external-dns not ignoring pre-existing DNS records it does not have ownership of. github.com/kubernetes-sigs/external-dns/issues/3715

On the solution:

  1. Cloudflare API: List DNS Records. developers.cloudflare.com/api/resources/dns/subresources/records/methods/list
  2. Terraform Registry: cloudflare_dns_record resource. registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/dns_record
  3. external-dns FAQ, registry, txt-owner-id, and policy flags. kubernetes-sigs.github.io/external-dns/v0.12.2/faq

Stuck on a tricky one?

If you have a DNS automation, reconciler, or ownership drift problem you would rather hand off, this is the kind of work I do. Message me and we can work through it together.

Contact me on LinkedIn

Did this stop a record from being overwritten?

If this saved you a scary reconcile diff or a record that kept flipping back and forth, you can buy me a coffee. It is the best way to keep these field notes free and growing.

Buy me a coffee on Ko-fi

Back to all DNS and Domains field notes