Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · DNS change auditor series ~6 min read

A DNS change auditor on AWS for a few dollars a month

DNS is the only part of a small business’s infrastructure where one wrong character takes everything down and nothing anywhere reports an error. The site is fine, the server is fine, the monitoring is green, and mail has been going to a hostname that no longer exists for eleven days. Nobody did anything reckless; somebody tidied up a record that turned out to matter. This post walks through a small system that makes every change visible within the hour.

white ceramic table with a white table cloth
Photo by Jan Zwarthoed on Unsplash

Key takeaways

  • An hourly snapshot of every record in every zone you rely on.
  • Every change is shown as a diff, in plain terms, to one nominated person.
  • Critical records — apex, MX, SPF, DMARC, NS — alarm immediately rather than in a digest.
  • It is read-only. There is no credential in it that can change a record.
  • Designed on AWS for about $1 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: zones snapshotted hourly, changes confirmed by a personThree boxes across the top sit outside the AWS account. On the left, Your zones: wherever they happen to be hosted. In the middle, Critical list: the records that must never move. On the right, Whoever owns DNS: the person who confirms a change or investigates it. Each connects by an arrow to the AWS account container below. Records as resolved flow down into the account. The critical list feeds in which records alarm. A question, was this meant, goes back out. Inside the AWS account are three components in a row. On the left, the Snapshotter, running hourly over every record, using a resolver rather than a provider API. In the middle, the Differ, which describes what changed in plain words. On the right, the Confirmer, which sends a digest or an alarm depending on what moved. A note at the bottom says the system is read-only and nothing in the account can change a DNS record.AWS ACCOUNTYour zoneswherever they are hostedCritical listwhat must never moveWhoever owns DNSconfirms or investigatesSnapshotterhourly, every record,resolver not APIDifferwhat changed,in plain wordsConfirmerdigest, or alarm,dependingrecords, asresolvedwhich ones alarmwas this meant?Read-only, always. Nothing in this account can change a DNS record.
Fig 1. Three things outside the account, three pieces inside it. The critical list in the middle is what separates a change that waits for a digest from one that rings a phone.
  • App integration
  • Networking
  • Security & identity
  • Analytics
  • People

Resolve, do not read the API

The obvious implementation reads your DNS provider’s API and diffs the zone file. It is easier, it gives you the TTLs and the comments, and it answers a slightly different question from the one that matters.

What matters is what the world sees. A zone can be correct in a provider’s console and wrong in resolution — a delegation that was never updated, a second provider still authoritative from a migration two years ago, a record shadowed by a wildcard. Resolving the records the way a mail server or a browser would is the only check that catches those, and it works identically regardless of who hosts the zone.

What runs hourly (the inside)

  • The snapshotter. Queries every record type you care about for every name you have listed, from a resolver, and stores the answers. Part 2 covers what to query when you do not have a zone file to enumerate from.
  • The differ. Compares this hour with last hour and describes the difference in terms a person can act on. “The MX record now points at mail.oldhost.example” rather than a unified diff of two text blobs.
  • The confirmer. Routes it. A change to an ordinary record goes into a daily digest with a one-tap “that was me”. A change to anything on the critical list alarms immediately, to two people.

One change, end to end

One DNS change from snapshot to confirmation, in five stagesA horizontal row of five boxes joined by arrows. Snapshot: every hour. Diff: against last hour. Critical: is it the apex, MX, SPF or nameservers. Alarm or digest: within minutes, or tomorrow. Confirmed: that was me, or not. A note says most changes are somebody's work and the point is that all of them are seen.ONE DNS CHANGE, END TO ENDSnapshotevery hourDiffagainst last hourCritical?apex, MX, SPF, NSAlarm or digestminutes, or tomorrowConfirmedthat was me, or notMost changes are somebody's work. The point is that all of them are seen.
Fig 2. The same system as one line. Nearly every change is legitimate, and the value is in the small number that are not being visible within the hour rather than in eleven days.
  • App integration
  • Machine learning
  • Analytics

In plain words

A developer is setting up a staging environment and adds a few records. Along the way they tidy what looks like a stale TXT record with a lot of odd syntax in it. It was the SPF record.

Nothing breaks. The site is fine, mail still sends, and the monitoring stays green, because SPF failure does not stop mail leaving — it changes what receiving servers do with it. Over the following fortnight, an increasing proportion of the business’s invoices and quotes land in junk folders, and the only symptom is customers saying they never got it. That is a genuinely difficult thing to diagnose from the outside and a trivial one to diagnose from a diff.

With this system the change is detected within the hour, it is on the critical list, and two people get a message: “TXT at example.com changed. Removed: v=spf1 include:... ~all. Nothing replaced it.” The developer says “that was me, I thought it was stale”, puts it back, and the whole incident is eleven minutes long instead of a fortnight.

Design rules that shaped every decision

  • Resolve rather than read an API. What the world sees is the only thing that matters.
  • Read-only, permanently. A system that can change DNS to fix DNS is a much more dangerous object.
  • Describe changes in plain words. A unified diff of two zone files is not something somebody reads at nine on a Tuesday.
  • Critical records alarm; everything else digests. Alarming on every CNAME trains people to ignore alarms.
  • “That was me” is one tap and is the expected answer. This is an audit trail, not an approval process.
  • Snapshot from more than one resolver. Propagation and split-horizon both produce false changes.

Why this shape

DNS changes are frequent, legitimate, and made by people who are usually right. A system that tried to gate them would be worked around within a week and would deserve to be. The gap is not control; it is that nobody currently knows a change happened unless they made it.

So this does exactly one thing: it makes every change visible, quickly, to one person who can recognise their own work in two seconds and will notice immediately when something is not their work. That is a much weaker control than approval and it catches the cases that actually happen.

The next four posts walk through each piece: how a zone gets enumerated and snapshotted, how a diff is described, why mail records get their own treatment, and how confirmation works. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts