Reconciler Provider API / Reconciliation
Duplicate record set falsely flagged as conflict
Your Terraform plan or CI script refuses to add a weighted or multivalue record, calling it a duplicate, even though the zone is set up exactly the way it should be. This happens because most homemade dedup logic only looks at a record's name and type, and never looks at the field that makes weighted, latency, failover, and multivalue routing possible in the first place: SetIdentifier. Here is why the zone is fine, and a script that fixes the check instead of the DNS.
The zone is almost always correct. The bug is in the reconciler, CI script, or Terraform-adjacent tool that decides what counts as a "duplicate" record. It keys its check on (name, type) only, so it sees two records that share a name and type, like two weighted A records for the same host, and calls the second one a conflict. Route 53 (and providers like it) allow this on purpose, using a separate field called SetIdentifier to tell the records apart. Fix the identity key to (name, type, SetIdentifier), or (name, type, content) for providers with no SetIdentifier concept, and the false positive goes away without touching the zone.
The problem in plain words
Weighted routing, latency routing, failover routing, geoproximity routing, and multivalue answer routing all work the same basic way: you create more than one record with the identical name and identical type, like several A records all named acme.example.com, and the DNS provider picks which one to hand back based on rules you set. Route 53 tells these records apart using a field called SetIdentifier, a label you choose, such as us-east-primary or us-east-secondary.
A dedup or reconciliation script that was written without this in mind checks whether a record already exists by comparing only the name and the type. Two records named acme.example.com with type A look identical to that check, even though one carries SetIdentifier: us-east-primary and the other carries SetIdentifier: us-east-secondary. The script throws an error, or a Terraform-adjacent tool refuses the change batch, and a completely legitimate weighted pair gets blocked from ever reaching the zone.
Why it happens
- The script or CI/Terraform-adjacent wrapper that diffs "intended" records against "live" records was written for the simple case of one record per name and type, and nobody added SetIdentifier to its comparison key when weighted or failover routing was introduced later.
- A plan that only changes
set_identifieron an existing record is a case that has tripped up automation before, since some tooling does not treat that field as significant enough to recompute an update. - For Cloudflare-style providers there is no SetIdentifier at all. Cloudflare natively allows multiple A or AAAA records with the identical name, so a client-side tool that dedups by name and type alone will refuse to add a second, different A record and call it a duplicate.
- Multivalue answer routing uses
MultiValueAnswer: trueplus a unique SetIdentifier per health-checked value instead of Weight, and a checker that only recognizes Weight as the "this is intentional" signal misses it too.
Route 53's own uniqueness rule for a resource record set is (Name, Type, SetIdentifier), not (Name, Type). Any reconciler, dedup script, or homemade change-batch validator that keys off fewer fields than the provider itself uses will eventually flag a completely valid routing policy as a conflict. The fix belongs in your tooling's identity key, not in the zone, which is usually already correct.
The fix, as a flow
Pull the live record sets from the provider and compare them to the intended list using the same identity key the provider itself uses. For Route 53 that is name, type, and SetIdentifier, treating a record with no SetIdentifier as belonging to the plain simple-routing group. For a provider with no SetIdentifier field, fall back to name, type, and the actual value (the IP or target) so two different, intentional values are never merged into one. Only records that are genuinely identical on every one of those fields count as a duplicate. Everything else is a legitimate second (or third) record in the set and should be allowed through.
How to fix it
Confirm the zone is actually fine
List the live record sets for the name and look for the SetIdentifier and Weight fields. If you see two or more entries, each with its own SetIdentifier, the zone is correct and the problem is downstream in your tooling.
aws route53 list-resource-record-sets \
--hosted-zone-id Z1D633PJN98FT9 \
--query "ResourceRecordSets[?Name=='acme.example.com.']"
# a correct weighted setup shows two+ entries, each with its own
# SetIdentifier and Weight, for example:
# { "Name": "acme.example.com.", "Type": "A",
# "SetIdentifier": "us-east-primary", "Weight": 100, ... }
# { "Name": "acme.example.com.", "Type": "A",
# "SetIdentifier": "us-east-secondary", "Weight": 50, ... }Confirm what the resolver actually returns
A legitimate weighted or multivalue setup returns different IPs on repeated queries, roughly in proportion to the weights, or several IPs at once for multivalue. That is expected behavior, not an error.
for i in 1 2 3 4 5; do dig +short A acme.example.com @8.8.8.8; done
# expect the two configured IPs to show up, roughly matching
# the weight ratio across repeated queriesFix the reconciler's identity key, not the zone
In whatever script diffs "intended" against "live" records before calling the provider's change API, replace a name-and-type-only key with one that includes SetIdentifier (falling back to content when SetIdentifier is absent).
{
"Name": "acme.example.com",
"Type": "A",
"SetIdentifier": "us-east-primary",
"Weight": 100,
"TTL": 60,
"ResourceRecords": [{ "Value": "192.0.2.10" }]
}
{
"Name": "acme.example.com",
"Type": "A",
"SetIdentifier": "us-east-secondary",
"Weight": 50,
"TTL": 60,
"ResourceRecords": [{ "Value": "192.0.2.11" }]
}
# multivalue answer routing: use MultiValueAnswer plus a unique
# SetIdentifier per health-checked value instead of Weight
# Cloudflare has no SetIdentifier concept. Stop deduping A/AAAA
# records by name+type in client tooling. Key on (name, type, content)
# instead, so two records are only "duplicates" when the target
# value is identical too.Re-submit the change
Once the identity key is fixed, submit the change batch again. It should apply cleanly with no InvalidChangeBatch error, since the two records were never really duplicates.
aws route53 change-resource-record-sets \
--hosted-zone-id Z1D633PJN98FT9 \
--change-batch file://weighted-change.json
# expect: "Status": "PENDING", no InvalidChangeBatch error
aws route53 get-change --id "$CHANGE_ID"
# poll until "Status": "INSYNC"How to check it worked
Confirm both weighted records are live as distinct entries, not one overwritten record, and that resolution still shows the expected distribution.
# two distinct entries, different SetIdentifier/Weight, not one overwritten record
aws route53 list-resource-record-sets \
--hosted-zone-id Z1D633PJN98FT9 \
--query "ResourceRecordSets[?Name=='acme.example.com.'].[SetIdentifier,Weight,ResourceRecords]"
# resolution still spreads across both IPs roughly by weight
for i in 1 2 3 4 5; do
dig +short A acme.example.com @ns-1234.awsdns-12.org
done
# Cloudflare-style multivalue: all intended IPs should be present
dig +short A www.example.comThe full code
Here is the complete checker in one file for each language. It reads records from the Cloudflare API (or, in comments, boto3 for Route 53), builds an identity key from name, type, and SetIdentifier or content, diffs against an intended list, and, only when you turn dry run off, applies the record that was wrongly blocked.
View this code on GitHub Full runnable folder with tests in the dns-fixes repo.
"""Detect and repair a false-positive duplicate record conflict caused by
a dedup/reconciler check that only keys on (name, type) instead of the
provider's real identity key of (name, type, set_identifier).
Safe by default. Set DRY_RUN=false to let it write.
Env vars:
DNS_DOMAIN the name to check, e.g. "acme.example.com"
CLOUDFLARE_API_TOKEN Cloudflare API token (only needed for repair)
CLOUDFLARE_ZONE_ID Cloudflare zone id (only needed for repair)
DRY_RUN default "true"; set to "false" to actually write
"""
import os
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("false_positive_duplicate_detection")
DNS_DOMAIN = os.environ.get("DNS_DOMAIN", "example.com")
CLOUDFLARE_API_TOKEN = os.environ.get("CLOUDFLARE_API_TOKEN", "")
CLOUDFLARE_ZONE_ID = os.environ.get("CLOUDFLARE_ZONE_ID", "")
DRY_RUN = os.environ.get("DRY_RUN", "true").lower() == "true"
CF_API = "https://api.cloudflare.com/client/v4"
def is_duplicate_record(existing: dict, candidate: dict) -> bool:
"""Pure decision function. No I/O.
existing/candidate keys: name (str, lowercased FQDN), type (str),
set_identifier (str|None), content (str, e.g. IP or target).
Returns True only if the records are truly the same record set.
Route53-style: identity key is (name, type, set_identifier).
Provider-without-set-id (e.g. Cloudflare A/AAAA multivalue): fall back
to (name, type, content) so distinct values are never merged.
"""
same_name = existing["name"].rstrip(".").lower() == candidate["name"].rstrip(".").lower()
same_type = existing["type"].upper() == candidate["type"].upper()
if not (same_name and same_type):
return False
if existing.get("set_identifier") or candidate.get("set_identifier"):
return existing.get("set_identifier") == candidate.get("set_identifier")
return existing["content"] == candidate["content"]
def find_real_conflicts(existing_records, intended_records):
"""Given the live zone and the intended record list, return only the
intended records that are true duplicates of something already live.
Everything else (a new SetIdentifier, a new value) is not a conflict
and should be allowed through.
"""
conflicts = []
for candidate in intended_records:
for existing in existing_records:
if is_duplicate_record(existing, candidate):
conflicts.append(candidate)
break
return conflicts
def list_zone_records(name=None):
"""List DNS records in the Cloudflare zone, optionally filtered by name."""
import requests
headers = {"Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}"}
params = {"per_page": 5000}
if name:
params["name"] = name
r = requests.get(
f"{CF_API}/zones/{CLOUDFLARE_ZONE_ID}/dns_records",
headers=headers, params=params, timeout=30,
)
r.raise_for_status()
return [
{
"name": rec["name"],
"type": rec["type"],
"content": rec["content"],
"set_identifier": None, # Cloudflare has no SetIdentifier concept
"id": rec["id"],
}
for rec in r.json()["result"]
]
# For Route 53 instead, lazily import boto3 and use:
# import boto3
# client = boto3.client("route53")
# resp = client.list_resource_record_sets(HostedZoneId=zone_id)
# for rrset in resp["ResourceRecordSets"]:
# set_identifier = rrset.get("SetIdentifier")
# ...
def apply_record(candidate):
"""Apply a genuinely new/changed record through the Cloudflare API
(POST for a new record, PUT for an update to an existing one).
"""
import requests
headers = {
"Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
"Content-Type": "application/json",
}
body = {
"type": candidate["type"],
"name": candidate["name"],
"content": candidate["content"],
"ttl": candidate.get("ttl", 300),
}
if DRY_RUN:
log.info("[dry run] would apply record %s", body)
return
requests.post(
f"{CF_API}/zones/{CLOUDFLARE_ZONE_ID}/dns_records",
headers=headers, json=body, timeout=30,
).raise_for_status()
log.info("Applied record %s", body)
# For Route 53 instead, this would call:
# client.change_resource_record_sets(
# HostedZoneId=zone_id,
# ChangeBatch={"Changes": [{"Action": "UPSERT", "ResourceRecordSet": rrset}]},
# )
def run():
if not (CLOUDFLARE_API_TOKEN and CLOUDFLARE_ZONE_ID):
log.warning("No Cloudflare credentials set. Nothing to check for %s.", DNS_DOMAIN)
return
existing_records = list_zone_records(DNS_DOMAIN)
# The "intended" list normally comes from source control (Terraform
# state, a records.yaml file, and so on). This is a stand-in example
# of a second weighted record that a bad (name, type)-only dedup
# check would have blocked.
intended_records = [
{
"name": DNS_DOMAIN,
"type": "A",
"set_identifier": "us-east-secondary",
"content": "192.0.2.11",
"ttl": 60,
},
]
real_conflicts = find_real_conflicts(existing_records, intended_records)
false_positives = [r for r in intended_records if r not in real_conflicts]
if false_positives:
log.info(
"%d record(s) were not real duplicates and will be applied.",
len(false_positives),
)
for record in false_positives:
apply_record(record)
if real_conflicts:
log.warning(
"%d record(s) are true duplicates (same name, type, and "
"set_identifier or content) and were skipped.",
len(real_conflicts),
)
log.info("Done.")
if __name__ == "__main__":
run()/**
* Detect and repair a false-positive duplicate record conflict caused by
* a dedup/reconciler check that only keys on (name, type) instead of the
* provider's real identity key of (name, type, set_identifier).
*
* Safe by default. Set DRY_RUN=false to let it write.
*
* Env vars:
* DNS_DOMAIN the name to check, e.g. "acme.example.com"
* CLOUDFLARE_API_TOKEN Cloudflare API token (only needed for repair)
* CLOUDFLARE_ZONE_ID Cloudflare zone id (only needed for repair)
* DRY_RUN default "true"; set to "false" to actually write
*/
import { pathToFileURL } from "node:url";
const DNS_DOMAIN = process.env.DNS_DOMAIN || "example.com";
const CLOUDFLARE_API_TOKEN = process.env.CLOUDFLARE_API_TOKEN || "";
const CLOUDFLARE_ZONE_ID = process.env.CLOUDFLARE_ZONE_ID || "";
const DRY_RUN = (process.env.DRY_RUN || "true").toLowerCase() === "true";
const CF_API = "https://api.cloudflare.com/client/v4";
export function isDuplicateRecord(existing, candidate) {
// Pure decision function. No I/O.
// existing/candidate keys: name (lowercased FQDN string), type (string),
// setIdentifier (string|null), content (string, e.g. IP or target).
// Returns true only if the records are truly the same record set.
// Route53-style: identity key is (name, type, setIdentifier).
// Provider-without-set-id (e.g. Cloudflare A/AAAA multivalue): fall back
// to (name, type, content) so distinct values are never merged.
const sameName = existing.name.replace(/\.$/, "").toLowerCase() ===
candidate.name.replace(/\.$/, "").toLowerCase();
const sameType = existing.type.toUpperCase() === candidate.type.toUpperCase();
if (!(sameName && sameType)) return false;
if (existing.setIdentifier || candidate.setIdentifier) {
return existing.setIdentifier === candidate.setIdentifier;
}
return existing.content === candidate.content;
}
export function findRealConflicts(existingRecords, intendedRecords) {
// Given the live zone and the intended record list, return only the
// intended records that are true duplicates of something already live.
// Everything else (a new setIdentifier, a new value) is not a conflict
// and should be allowed through.
const conflicts = [];
for (const candidate of intendedRecords) {
if (existingRecords.some((existing) => isDuplicateRecord(existing, candidate))) {
conflicts.push(candidate);
}
}
return conflicts;
}
async function listZoneRecords(name) {
const headers = { Authorization: `Bearer ${CLOUDFLARE_API_TOKEN}` };
const params = new URLSearchParams({ per_page: "5000" });
if (name) params.set("name", name);
const res = await fetch(
`${CF_API}/zones/${CLOUDFLARE_ZONE_ID}/dns_records?${params.toString()}`,
{ headers },
);
if (!res.ok) throw new Error(`Cloudflare list returned ${res.status}`);
const body = await res.json();
return body.result.map((rec) => ({
name: rec.name,
type: rec.type,
content: rec.content,
setIdentifier: null, // Cloudflare has no SetIdentifier concept
id: rec.id,
}));
// For Route 53 instead, this would use the AWS SDK v3
// @aws-sdk/client-route-53-domains style client and read
// rrset.SetIdentifier off each ResourceRecordSet.
}
async function applyRecord(candidate) {
const headers = {
Authorization: `Bearer ${CLOUDFLARE_API_TOKEN}`,
"Content-Type": "application/json",
};
const body = {
type: candidate.type,
name: candidate.name,
content: candidate.content,
ttl: candidate.ttl || 300,
};
if (DRY_RUN) {
console.log(`[dry run] would apply record ${JSON.stringify(body)}`);
return;
}
const res = await fetch(`${CF_API}/zones/${CLOUDFLARE_ZONE_ID}/dns_records`, {
method: "POST",
headers,
body: JSON.stringify(body),
});
if (!res.ok) throw new Error(`Cloudflare create returned ${res.status}`);
console.log(`Applied record ${JSON.stringify(body)}`);
}
async function run() {
if (!(CLOUDFLARE_API_TOKEN && CLOUDFLARE_ZONE_ID)) {
console.warn(`No Cloudflare credentials set. Nothing to check for ${DNS_DOMAIN}.`);
return;
}
const existingRecords = await listZoneRecords(DNS_DOMAIN);
// The "intended" list normally comes from source control. This is a
// stand-in example of a second weighted record that a bad
// (name, type)-only dedup check would have blocked.
const intendedRecords = [
{
name: DNS_DOMAIN,
type: "A",
setIdentifier: "us-east-secondary",
content: "192.0.2.11",
ttl: 60,
},
];
const realConflicts = findRealConflicts(existingRecords, intendedRecords);
const falsePositives = intendedRecords.filter((r) => !realConflicts.includes(r));
if (falsePositives.length > 0) {
console.log(`${falsePositives.length} record(s) were not real duplicates and will be applied.`);
}
for (const record of falsePositives) {
await applyRecord(record);
}
if (realConflicts.length > 0) {
console.warn(
`${realConflicts.length} record(s) are true duplicates (same name, type, and ` +
`setIdentifier or content) and were skipped.`,
);
}
console.log("Done.");
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
run().catch((err) => {
console.error(err);
process.exit(1);
});
}Add a test
The identity check is the part most worth testing, because it decides whether a legitimate change gets blocked. Because is_duplicate_record is pure, the test needs no network and no provider account. It just feeds in plain dicts or objects and checks the result.
from false_positive_duplicate_detection import is_duplicate_record
def rec(name="acme.example.com", type_="A", set_identifier=None, content="192.0.2.10"):
return {"name": name, "type": type_, "set_identifier": set_identifier, "content": content}
def test_weighted_records_with_different_set_identifier_are_not_duplicates():
existing = rec(set_identifier="us-east-primary", content="192.0.2.10")
candidate = rec(set_identifier="us-east-secondary", content="192.0.2.11")
assert is_duplicate_record(existing, candidate) is False
def test_same_set_identifier_is_a_true_duplicate():
existing = rec(set_identifier="us-east-primary", content="192.0.2.10")
candidate = rec(set_identifier="us-east-primary", content="192.0.2.10")
assert is_duplicate_record(existing, candidate) is True
def test_no_set_identifier_falls_back_to_content_cloudflare_style():
existing = rec(set_identifier=None, content="192.0.2.10")
candidate = rec(set_identifier=None, content="192.0.2.11")
assert is_duplicate_record(existing, candidate) is False
def test_no_set_identifier_same_content_is_a_true_duplicate():
existing = rec(set_identifier=None, content="192.0.2.10")
candidate = rec(set_identifier=None, content="192.0.2.10")
assert is_duplicate_record(existing, candidate) is True
def test_different_name_is_never_a_duplicate():
existing = rec(name="acme.example.com")
candidate = rec(name="other.example.com")
assert is_duplicate_record(existing, candidate) is Falseimport { test } from "node:test";
import assert from "node:assert/strict";
import { isDuplicateRecord } from "./false-positive-duplicate-detection.js";
const rec = ({ name = "acme.example.com", type = "A", setIdentifier = null, content = "192.0.2.10" } = {}) =>
({ name, type, setIdentifier, content });
test("weighted records with different setIdentifier are not duplicates", () => {
const existing = rec({ setIdentifier: "us-east-primary", content: "192.0.2.10" });
const candidate = rec({ setIdentifier: "us-east-secondary", content: "192.0.2.11" });
assert.equal(isDuplicateRecord(existing, candidate), false);
});
test("same setIdentifier is a true duplicate", () => {
const existing = rec({ setIdentifier: "us-east-primary", content: "192.0.2.10" });
const candidate = rec({ setIdentifier: "us-east-primary", content: "192.0.2.10" });
assert.equal(isDuplicateRecord(existing, candidate), true);
});
test("no setIdentifier falls back to content, Cloudflare style", () => {
const existing = rec({ setIdentifier: null, content: "192.0.2.10" });
const candidate = rec({ setIdentifier: null, content: "192.0.2.11" });
assert.equal(isDuplicateRecord(existing, candidate), false);
});
test("no setIdentifier same content is a true duplicate", () => {
const existing = rec({ setIdentifier: null, content: "192.0.2.10" });
const candidate = rec({ setIdentifier: null, content: "192.0.2.10" });
assert.equal(isDuplicateRecord(existing, candidate), true);
});
test("different name is never a duplicate", () => {
const existing = rec({ name: "acme.example.com" });
const candidate = rec({ name: "other.example.com" });
assert.equal(isDuplicateRecord(existing, candidate), false);
});Case studies
The failover pair that Terraform would not add
A team stood up a primary/secondary failover pair for a payments endpoint, both records named the same and typed A, each with its own SetIdentifier and health check. A homemade guard script that ran before every terraform apply, meant to stop accidental duplicate records, keyed its check on name and type only and rejected the plan with a "duplicate record" error, blocking the failover rollout the night before a planned maintenance window.
Listing the live zone showed the primary record already existed with its own SetIdentifier, and the secondary was a genuinely new SetIdentifier value, not a duplicate at all. Updating the guard script's key to include SetIdentifier let the plan apply cleanly, and the failover pair went live on schedule.
The multivalue answer that looked like a copy-paste mistake
A CI pipeline that pushed DNS changes from a records.yaml file included a new multivalue-answer record for a service discovery hostname, three A records with the same name, each with MultiValueAnswer: true and its own SetIdentifier tied to a health check. The pipeline's diff step, written before multivalue routing was in use, treated the second and third entries as duplicates of the first and silently dropped them from the change batch.
Only one of the three intended IPs ever reached Route 53. The team traced it by comparing the intended file to list-resource-record-sets output and noticed the missing SetIdentifiers. Fixing the diff step's key to include SetIdentifier let all three health-checked values through.
After the fix, a weighted, latency, failover, geoproximity, or multivalue record set applies on the first try, with no InvalidChangeBatch error and no silently dropped entries. list-resource-record-sets shows every intended SetIdentifier present with its correct Weight, and repeated dig queries return the expected distribution of answers. Genuine duplicates, records that match on name, type, and SetIdentifier (or content) exactly, are still caught and blocked, so the fix does not open the door to real accidental duplication.
FAQ
Why does my weighted or multivalue record get rejected as a duplicate?
A reconciler or dedup script is grouping records only by name and type. Route 53 and similar providers allow more than one record with the same name and type as long as each has a distinct SetIdentifier, which is how weighted, latency, failover, geoproximity, and multivalue routing all work. When the script ignores SetIdentifier it sees two records with the same name and type and calls the second one a duplicate, even though the zone is correct.
Is the DNS zone actually broken when I see this error?
Usually not. The zone is almost always fine. List the live records and check whether each one carries its own SetIdentifier and Weight (or MultiValueAnswer flag). If they do, the zone is correct and the problem is the identity key your automation uses to decide what counts as a duplicate.
How do I fix the false positive without breaking real duplicate detection?
Change the uniqueness key from (name, type) to (name, type, SetIdentifier). Treat a missing SetIdentifier as belonging to the plain simple-routing group. For providers with no SetIdentifier concept, such as Cloudflare's plain A/AAAA records, key on (name, type, content) instead so records are only merged when the name, type, and target value are all identical.
Related field notes
Citations
On the problem:
- Amazon Route 53 API Reference: ResourceRecordSet, SetIdentifier semantics. docs.aws.amazon.com/Route53
- hashicorp/terraform-provider-aws issue #36513: incorrect duplicate Route 53 record detection. github.com/hashicorp/terraform-provider-aws
- Cloudflare DNS docs: cannot add DNS records with the same name. developers.cloudflare.com/dns
On the solution:
- Amazon Route 53 API Reference: ChangeResourceRecordSets. docs.aws.amazon.com/Route53
- Amazon Route 53 Developer Guide: values specific for weighted records. docs.aws.amazon.com/Route53
- hashicorp/terraform-provider-aws issue #7998: aws_route53_record plans that only change set_identifier do not apply correctly. github.com/hashicorp/terraform-provider-aws
Stuck on a tricky one?
If you have a DNS, domain, or certificate problem you would rather hand off, this is the kind of work I do. Message me and we can work through it together.
Did this unblock your rollout?
If this saved you a blocked deploy or a confusing InvalidChangeBatch error, you can buy me a coffee. It is the best way to keep these field notes free and growing.
Buy me a coffee on Ko-fi