Skip to content
allanninal.dev/build
Collaborate

Part 1 of 7 · Search rank reporter series ~6 min read

A search rank reporter on AWS for a few dollars a month

Search data has the opposite problem from most business data: there is too much of it and it is too detailed. Nine hundred query rows a week, most of them with four impressions, all of them moving up and down for reasons nobody can see. It is genuinely informative and completely unreadable, so it gets exported to a spreadsheet once and never looked at again. This post walks through a small system that turns it into six lines.

man buying coffee on counter
Photo by Joshua Rodriguez on Unsplash

Key takeaways

  • It reads your own performance data. Nothing is scraped and nothing is simulated.
  • Queries are grouped into themes you define, so the report matches how you think.
  • Movement is reported only when the query group has enough volume for it to be real.
  • The report names the page responsible, because that is what somebody can change.
  • Designed on AWS for about $1 a month.

The whole system on one page

Before any code, here is the shape of what we are designing.

System: search performance data grouped into themes and reportedThree boxes across the top sit outside the AWS account. On the left, Search performance: your own data, read through an API. In the middle, Theme rules: the mapping from queries to themes. On the right, Whoever owns the site: the person who receives six lines a week. Each connects by an arrow to the AWS account container below. Impressions, clicks and position flow down into the account. The theme rules feed in what counts as what. Six lines and the pages responsible go back out. Inside the AWS account are three components in a row. On the left, the Fetcher, running weekly, pulling data by query and by page and storing it raw. In the middle, the Grouper, turning nine hundred rows into six themes. On the right, the Reporter, surfacing movement worth reading together with the pages. A note at the bottom says everything reported is measured and nothing here estimates a ranking.AWS ACCOUNTSearch performanceyour own data, by APITheme rulesqueries to themesWhoever owns the sitesix lines a weekFetcherweekly, query by page,stored rawGrouper900 rows intosix themesReportermovement worthreading, with pagesimpressions,clicks, positionwhat counts aswhatsix lines and thepagesEverything reported is measured. Nothing here estimates a ranking.
Fig 1. Three things outside the account, three pieces inside it. The theme rules in the middle are the only part that requires judgement, and they are a sheet.
  • Database
  • Analytics
  • People

Measured, not simulated

There is an entire industry built on typing your keywords into a search engine from a datacentre and recording what comes back. It produces a number, and the number is not what your customers saw: results are personalised, localised, device-dependent and increasingly assembled per query.

Your own performance data is the average position across the impressions that actually happened, to real people, in the places they were. It is the better number by some distance, it costs nothing, and getting it does not involve breaking anybody’s terms of service.

What runs weekly (the inside)

  • The fetcher. Pulls last week’s data by query and by page, and stores it raw before anything is done to it. Part 2 covers the sampling and the delay, both of which surprise people.
  • The grouper. Maps query rows onto themes using rules you write. Part 3 is about why the rules are yours rather than generated, and why grouping is the whole product.
  • The reporter. Compares this period with the last, decides what movement is large enough to be real given the volume behind it, and names the pages that carry each theme.

One theme, end to end

One week of search data from fetch to report, in five stagesA horizontal row of five boxes joined by arrows. Fetched: nine hundred query rows. Grouped: into six themes. Compared: against the last eight weeks. Volume check: is the movement real. Reported: with the pages responsible. A note says the fourth step removes most of what a raw export would have shown you.ONE WEEK, END TO ENDFetched900 query rowsGroupedinto 6 themesComparedto the last 8 weeksVolume checkis the move real?Reportedwith the pagesThe fourth step removes most of what a raw export would have shown you.
Fig 2. The same system as one line. The volume check is what stops a theme with eleven impressions appearing next to one with four thousand.

In plain words

A plumbing firm’s site gets about nine hundred distinct search queries a week. Grouped, they fall into six themes that match what the business actually does: emergency callouts, boiler installation, bathroom fitting, commercial contracts, the company name, and everything else.

This week, emergency callouts is down. Average position has gone from 4.1 to 7.8 and clicks are down sixty per cent, on four thousand impressions — which is far too much volume for that to be noise. The theme is carried almost entirely by one page, and the report says so: “emergency callouts: position 4.1 to 7.8, clicks down 60%, on 4,100 impressions. 92% of this theme lands on /emergency-plumber.”

That is a report somebody acts on within the hour, and it takes about nine seconds to read. The raw export containing the same information is a hundred and forty rows of queries containing the word emergency, each with its own small movement, and it communicates none of it.

Design rules that shaped every decision

  • Use measured data, never simulated positions. Scraped rankings are a different and worse number.
  • Group by theme, always. Query-level reporting is unreadable and mostly noise.
  • Volume gates everything. A move on eleven impressions is not a move.
  • Name the page, not the query. Pages are what somebody can change.
  • Correlate, never claim causation. Search movement has causes no data you hold can see.
  • Report the same six themes every week, so the report is comparable rather than a fresh surprise.

Why this shape

The data is already available, free, and accurate. The reason nobody uses it is entirely about granularity: it arrives at the level of individual queries, and businesses think at the level of what they sell. Every intermediate tool tries to bridge that with dashboards, and dashboards have the same problem as spreadsheets — somebody has to go and look.

So this design does one transformation, weekly, and pushes six lines. The transformation is grouping, the gate is volume, and the addition is naming the page. None of it is sophisticated and all of it is the difference between data that exists and data that gets used.

The next four posts walk through each piece: how the data is fetched, how queries are grouped, how a real movement is told from noise, and what the report says. One diagram per post, a cost breakdown, and an engineering reference at the end.

All posts