Skip to content
allanninal.dev/build
Collaborate

Part 5 of 7 · Capacity forecaster series ~5 min read

How a forecast gets scored afterwards

A forecast that is never compared to what happened is an opinion that was written down, and the difference between the two is a year of quiet learning that nobody collects.

Key takeaways

  • Store every forecast at the moment it was made, immutably.
  • Score by horizon: a four-week forecast and a twenty-week one are different skills.
  • Systematic bias is more useful to know about than random error.
  • The forecast that changed a decision cannot be scored fairly, and say so.
  • Correct the bias in the method, not by adjusting individual forecasts.

Store it as it was

How a forecast is stored so it can be scored laterA vertical chain of five steps entered by a box labelled A forecast is produced, weekly. Step one writes it down, every week and every horizon. Step two never updates it, appending a new one instead. Step three waits until the week arrives. Step four records what happened, actual demand and output. Step five scores by horizon: four weeks out, twelve, and twenty-six. A note says the second box is the whole discipline, and a revised forecast is a different forecast.AWS ACCOUNTA forecast is producedweeklyWrite it downevery week, every horizonNever update itappend a new one insteadWaituntil the week arrivesRecord what happenedactual demand and outputScore by horizon4 weeks out, 12, 26The second box is the whole discipline. A revised forecast is a different forecast.
Fig 1. How forecasts become scoreable. Storing the forecast as it was made, rather than the latest version, is the only step that requires any discipline.
  • Database
  • Security & identity
  • Management
  • Analytics

By horizon, always

Forecasting four weeks ahead and twenty-six weeks ahead are different tasks with different error rates, and combining them into one accuracy figure produces a number that describes neither.

Scored separately, the pattern is usually clear: near-term forecasts are good because the order book covers most of the demand, and long-term forecasts are good or bad depending on how well the seasonal pattern holds. Knowing which is which tells you where to spend effort.

Bias beats noise

Systematic bias and random error in forecasts at three horizonsA stacked bar chart with three bars in per cent. Two series: systematic under-forecast in red, and random error in grey. Four weeks out shows two per cent bias and six per cent noise. Twelve weeks out shows nine per cent bias and eleven per cent noise. Twenty-six weeks out shows fourteen per cent bias and nineteen per cent noise. A note says the red band is fixable and the grey band mostly is not.010203040~84 weeks out~2012 weeks out~3326 weeks outSystematic under-forecast, %Random error, %The red band is fixable. The grey band mostly is not.
Fig 2. A year of forecasts scored by horizon. The systematic component is the one worth acting on and it is invisible in a single forecast.

A consistent under-forecast at longer horizons is the most common finding and it usually has one cause: the late-arriving work from Part 3 is being underestimated, because the uplift was set from intuition rather than measured.

That is a correctable method problem. Adjusting the uplift and watching the bias fall over the next two quarters is a satisfying and unusual thing to be able to do, and it is only possible because the forecasts were stored.

Correct the method, not the forecast

The wrong response to a known bias is for whoever reads the forecast to mentally add fifteen per cent, because that adjustment is undocumented, inconsistent, and disappears when the person does. The right response is to change the uplift in the method and record why.

The forecast that changed the outcome

Why some forecasts cannot be scored fairly

  • The forecast said January would be 40 hours short.
  • So two people were hired and two deliveries were rescheduled.
  • January was fine. The forecast was wrong, by any naive scoring.
  • It was also correct and it worked, which is the entire purpose.
  • Record the actions taken against the forecast, so this case is visible rather than counted as an error.
  • Score demand, not the shortfall. Demand is largely unaffected by your response; capacity is not.

The last line is the practical resolution. Forecast accuracy should be measured on the demand side, where the forecast does not usually influence the outcome, and the capacity side should be recorded as a decision log rather than scored.

Without that separation a forecasting system that is working well appears to be performing badly, which is a reliable way for it to be abandoned.

What the annual review says

Once a year, half a page

  • Forecasts made: 52, each covering 26 weeks.
  • Demand accuracy at 4 weeks: within 8% on 46 of 52.
  • Demand accuracy at 26 weeks: within 20% on 31 of 52.
  • Systematic bias: under-forecasting by 9% at 12 weeks, corrected in June by raising the late-arrival uplift from 20% to 27%.
  • Shortfalls predicted: 6. Acted on: 5. Occurred anyway: 1.
  • Shortfalls that occurred with no warning: 2, both from a customer bringing a project forward.

The last line is the most valuable one on the page and the least comfortable. Shortfalls that arrived with no warning are what the system exists to prevent, and two of them in a year with an identified common cause is a finding about the order book rather than about the forecast.

Next: what all of this costs to run.

All posts