Skip to content
allanninal.dev/build
Collaborate

Part 5 of 7 · DNS change auditor series ~5 min read

How a DNS change gets confirmed

The last piece is the cheapest and it determines whether any of the rest gets read. This is an audit trail rather than an approval process, so confirming a change has to cost about two seconds, and the answer is nearly always yes.

Key takeaways

  • “That was me” is one tap and is the expected answer.
  • Ordinary changes go into a daily digest, not an alarm.
  • An unconfirmed critical change escalates within hours, to a second person.
  • Confirmation is recorded, so the log answers who knew about what and when.
  • If more than a couple of changes a month alarm, the critical list is too broad.

Two channels

How DNS changes are confirmed through two channelsThree boxes across the top outside the AWS account. Whoever owns DNS, who confirms. A second person, involved only if a change is unconfirmed. And the log, which records every change forever. Inside the account, three components. The Alarm path, for critical records, within minutes. The Digest path, for everything else, once a day. And the Escalator, for an unconfirmed critical change after four hours. Arrows show the owner replying that was me, a second person being reached after four hours, and everything reaching the log whether confirmed or not. A note says the log records every change whether or not anybody confirmed it.AWS ACCOUNTWhoever owns DNSconfirmsA second persononly if unconfirmedThe logevery change, foreverAlarm pathcritical records,within minutesDigest patheverything else,once a dayEscalatorunconfirmed critical,after four hoursthat was meafter four hourseverything,confirmed or notThe log records every change whether or not anybody confirmed it.
Fig 1. The two notification channels and the escalation between them. Everything reaches the log regardless; the channels only decide how quickly a person is asked.
  • Management
  • Analytics
  • People

The daily digest

Yesterday’s changes

  • 3 changes, all in staging.example.com. Added: api-v2, worker-2. Modified: api CNAME now points at the new load balancer.
  • 1 change at example.com. TXT _acme-challenge removed — this usually follows a certificate being issued.
  • SPF lookups: 7 of 10. Unchanged from last week.
  • Nothing on the critical list changed.
  • One tap: all of these were expected.

Four lines and one button. The last line is the one that gets read and it is the only one that matters on a normal day — and its absence on an abnormal day is far more noticeable than any amount of red.

Why not alarm on everything

Because a business doing ordinary work generates several DNS changes a week and none of them is an emergency. Alarming on all of them produces a channel people mute within a fortnight, and a muted channel does not carry the MX change either.

Unconfirmed critical changes

The interesting case is a critical record changing and nobody saying it was them. That is not necessarily bad — the person who did it may be driving — but it is the exact shape of the situation the system exists for.

How an unconfirmed critical DNS change escalatesA horizontal row of five boxes. MX changed: an alarm is sent. No confirmation: after four hours. Second person: told, with the diff. Still nothing: after eight hours. Everyone on the list: told, with what the record points at now. A note says usually somebody was busy, and the ladder is sized for the time it is not.AN UNCONFIRMED CRITICAL CHANGEMX changedalarm sentNo confirmationafter 4 hoursSecond personwith the diffStill nothingafter 8 hoursEveryone on the listwith what it points at nowUsually somebody was busy. The ladder is sized for the time it is not.
Fig 2. What happens when a critical change goes unconfirmed. The ladder is short because the situations it is sized for are measured in hours.
  • App integration
  • Security & identity
  • Management
  • People

Is the critical list right

One number tells you: how many alarms fired last month. The target is one or two. Zero for several months running probably means the list is too narrow — a business genuinely does change its DNS — and more than about four a month means it is too broad and the alarms are becoming routine.

The most common over-inclusion is putting every A record on the critical list. The apex matters; staging-api-3 does not, and including it means somebody gets a phone alert every time a developer does their job.

What the log answers

Every change, its diff, when it was seen, who confirmed it, and when. Twelve months of that answers a question that is otherwise unanswerable: “when did this record become wrong, and did anybody know?” It is a few thousand small rows and it costs nothing.

Next: what all of this costs to run.

All posts