Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Uptime status poster series ~6 min read

An uptime status poster on AWS for a few dollars a month

There are two ways a status page fails and most small businesses manage both. It goes down with the thing it monitors, so during the one hour it exists for it shows nothing. And it reports that the server is responding while customers cannot check out, because it pings a home page. This post walks through a small system built to avoid exactly those two things.

a person riding a motorcycle down a street
Photo by Grab on Unsplash

Key takeaways

  • Check journeys, not hosts. A 200 from the home page says almost nothing.
  • Check from three regions, and require agreement before calling anything down.
  • The status page is hosted on infrastructure that shares nothing with your site.
  • Automated status publishes itself; the human incident note is drafted and published by a person.
  • Designed on AWS for about $3 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: journeys checked from three regions, status published independentlyThree boxes across the top sit outside the AWS account. On the left, Your journeys: log in, search and check out. In the middle, Three regions checking independently. On the right, Customers and you: the people who read the page and publish the note. Each connects by an arrow to the AWS account container below. Real requests flow down into the account. Three regional opinions feed in. Status and a note go back out. Inside the AWS account are three components in a row. On the left, the Checkers, one per region running every minute. In the middle, the Decider, which requires agreement and then consecutive failures. On the right, the Publisher, which writes a static page on separate infrastructure. A note at the bottom says nothing in the publishing path depends on anything the checks are checking.AWS ACCOUNTYour journeyslog in, search,check outThree regionschecking independentlyCustomers + youread the page,publish the noteCheckersone per region,every minuteDecideragreement, thenconsecutive failuresPublishera static page,on separate infrastructurereal requeststhree opinionsstatus, and a noteNothing in the publishing path depends on anything the checks are checking.
Fig 1. Three things outside the account, three pieces inside it. The note at the bottom is the entire architectural requirement, and it constrains where every piece can live.
  • Networking
  • Management
  • Analytics
  • Front-end & mobile
  • People
  • Outside AWS

The independence requirement

A status page that lives on the same server, behind the same load balancer, in the same region, on the same DNS as your product is not a status page. It is a second copy of the thing that is broken.

So the requirement is stated as a rule rather than a preference: the publishing path shares nothing with the monitored path. Different region. Different bucket. Ideally a different domain or at minimum a subdomain whose DNS could survive the main zone being the problem. Part 4 works through what that means concretely and where the honest compromises are.

What runs every minute (the inside)

  • The checkers. One per region, running the journeys as real requests: a login that authenticates, a search that returns results, a checkout that reaches the payment step. Part 2 covers what a journey check is and what it must not do.
  • The decider. Takes three regional opinions and decides. Agreement first, then consecutive failures, then a state change. Part 3 is entirely about not calling an outage because one region had a bad minute.
  • The publisher. Writes a static page to a bucket in a different region behind its own distribution. The page is plain HTML with no JavaScript, because a status page has to render on a phone on a train.

One incident, end to end

One incident from first failed check to published note, in five stagesA horizontal row of five boxes joined by arrows. Checked: three regions, every minute. Two fail: one region still reports fine. All three fail: twice running. Published: the status changes itself. Note added: by a person. A note says steps two and three are about three minutes apart, and that is the deliberate delay.ONE INCIDENT, END TO ENDChecked3 regions, every minuteTwo failone region still fineAll three failtwice runningPublishedstatus changes itselfNote addedby a personSteps two and three are about three minutes apart. That is the deliberate delay.
Fig 2. The same system as one line. The gap between the first failure and publishing is the price of not publishing an outage that was a network blip.
  • App integration
  • Management
  • People
  • Outside AWS

In plain words

At 14:06 the checkout journey fails from Ireland and Frankfurt. London still succeeds, so nothing is published: two out of three is a disagreement, and disagreement usually means a network problem between a checker and you rather than an outage.

At 14:07 London fails too. All three agree, but one round of agreement is not enough. At 14:08 all three fail again, and the status page changes to “Checkout: degraded” automatically. Total elapsed time from the first symptom: two minutes.

At the same moment a message goes to whoever is on call with a drafted note: “Checkout is currently failing. We are looking into it.” They read it, agree, and tap publish, and now the status page says something a customer can understand rather than just a red dot. The whole sequence took four minutes and the customer emailing to ask whether it is just them has somewhere to look.

Design rules that shaped every decision

  • The publishing path shares nothing with the monitored path. That constrains everything else.
  • Check journeys, not hosts. A working home page is not evidence of a working business.
  • Require agreement across regions. One checker cannot distinguish your outage from its own.
  • Two consecutive agreeing failures. One is a blip and publishing it is worse than waiting.
  • The status changes itself; the sentence customers read is published by a person.
  • The page is static HTML, no JavaScript. It has to render badly on a bad connection.

Why this shape

Uptime monitoring is a solved problem with a dozen good hosted options, and for most businesses buying one is the right answer. The reason to build a small one is the status page rather than the monitoring: a hosted checker tells you, and a status page tells your customers, and the second is the thing that reduces support load during an outage.

So the design puts almost all of its care into the publishing path being independent and the page being readable, and treats the checking itself as the straightforward part. That is the opposite emphasis from most monitoring tools, and it follows directly from asking what the thing is for.

The next four posts walk through each piece: what a journey check actually does, how a check becomes an outage, how the page is served independently, and how an incident gets closed. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts