Skip to content
allanninal.dev/build
Collaborate

Part 2 of 7 · Search rank reporter series ~5 min read

How search data gets fetched

Fetching search performance data is an API call, and there are three properties of that data that catch everybody out the first time. None is a problem once you know about it, and all three will make your numbers wrong if you do not.

Key takeaways

  • The data lags by two to three days. A weekly fetch must not ask for yesterday.
  • Row limits truncate silently. Paginate, and check whether you hit the limit.
  • Long-tail and rare queries are withheld entirely, so totals will not reconcile.
  • Fetch by query and by page separately; the combination is not always available.
  • Store the raw response before grouping, so a rule change can be applied to history.

Three properties that surprise people

How a week of search performance data is fetchedA vertical chain of five steps entered by a box labelled Weekly fetch, for a completed week. Step one asks whether the data is settled, allowing three days; if not it exits to Wait, because the numbers are still moving. Step two fetches by query, paginated, against a row limit of twenty-five thousand per request. Step three asks whether the limit was hit, checking rather than assuming; if so it exits to Paginate again until a short page returns. Step four fetches by page as a separate request. Step five stores the raw response before any grouping. A note says a response exactly at the row limit is almost certainly truncated rather than complete.AWS ACCOUNTWeekly fetchfor a completed weekIs the data settled?allow three daysWaitnumbers still movingnoFetch by querypaginatedRow limit25,000 per requestHit the limit?check, do not assumePaginate againuntil shortyesFetch by pagea separate requestStore rawbefore any groupingA response exactly at the row limit is almost certainly truncated, not complete.
Fig 1. How one week is fetched. The truncation check is the one most implementations skip, and it silently loses the long tail that grouping most depends on.
  • Database
  • App integration
  • Management
  • Analytics
  • Front-end & mobile

The delay

Search performance data is not final when it first appears. Numbers for the last two or three days keep moving as data is processed, and a fetch that includes yesterday will produce a figure that is different if you run the same fetch again on Friday.

That produces the specific and maddening failure where a weekly report contradicts itself: this week’s number for last week is not the number you reported last week. So the fetch always asks for a week that ended at least three days ago, and the report says which week it covers.

Row limits

A request returns at most a fixed number of rows, and there is no flag in the response saying “there was more”. A response containing exactly the limit is almost certainly truncated; one containing fewer is complete. Checking that and paginating is three lines and its absence silently discards the long tail, which for a site with any content is most of the queries.

Withheld queries

Rare queries are not returned at all, for privacy reasons that are entirely reasonable. The practical consequence is that summing the query rows gives a smaller total than the site-level total, sometimes much smaller.

That is not an error to be reconciled away. The report states both numbers when they differ materially, because a business whose long tail is forty per cent of its impressions should know that its themed report covers sixty per cent of reality.

Store raw first

Why raw search rows are stored before groupingA horizontal row of five boxes. Raw stored: every row, as returned. Rules change: a new theme is added. Regroup history: all of it. Comparable: the new theme has old data behind it. Without raw: the new theme starts at zero. A note says theme rules change constantly, and keeping raw means history changes with them.WHY THE RAW RESPONSE IS KEPTRaw storedevery row, as returnedRules changea new theme addedRegroup historyall of itComparablenew theme, old dataWithout rawthe new theme starts at zeroTheme rules change constantly. Keeping raw means history changes with them.
Fig 2. Why raw rows are stored before grouping. Theme rules get refined every few weeks, and each refinement is worthless if it cannot be applied backwards.
  • Database
  • Management
  • Analytics

This is the single most valuable structural decision in the system and it costs a few megabytes a year. Theme rules get refined constantly — a new service line, a realisation that two themes should be one — and a system that only stores grouped totals has to start each new theme from the day it was created.

With raw rows kept, adding a theme in August produces eight months of history for it immediately, which is the difference between a rule change being cheap and being a decision.

Next: how the grouping works.

All posts