Skip to content
allanninal.dev/build
Collaborate

Part 4 of 7 · Exit interview collector series ~5 min read

How themes get found

Six free-text answers from each of a handful of leavers is not much data, which is both the constraint and the reason this works. There is no statistics here; there is grouping, a threshold, and one comparison that is unique to asking twice.

Key takeaways

  • Themes come from a fixed list you maintain, not from free clustering.
  • Nothing is reported below three people. Two is a coincidence.
  • A theme carries a count and a direction, never a quote.
  • The comparison between ask one and ask two is the finding nobody else has.
  • An answer that matches no theme is counted as unmatched, which is itself a signal.

A fixed theme list

The instinct is to let a model cluster free text and name the clusters. It produces different groupings every time it runs, which makes trends impossible, and it occasionally names a cluster in a way that is effectively a quote.

So the themes are a list you maintain — pay, scheduling, workload, management, progression, environment, commute, a better offer — and the model’s job is classification into that list, or none. Same input, same output, every time, and trends over quarters mean something.

How free-text exit answers become reportable themesA vertical chain of five steps entered by a box labelled Six answers, from one response. Step one classifies each answer with a single Bedrock call against your own fixed theme list. Step two asks whether any themes matched; none exits to Count as unmatched, which is a signal in itself. Step three increments the counts per theme per quarter in an aggregate table holding counts only. Step four asks whether three or more people have raised it across the window; fewer exits to Hold it, not yet a theme. Step five is Reportable, carrying a count and a direction. A note says the text is never stored on the aggregate and only counts cross into the report.AWS ACCOUNTSix answersone responseClassify each answerone Bedrock callTheme listyours, fixedAny themes matched?Count as unmatcheda signal in itselfnoneIncrement the countsper theme, per quarterAggregate tablecounts onlyThree or more people?across the windowHold itnot yet a themenoReportablea count and a directionThe text is never stored on the aggregate. Only counts cross into the report.
Fig 1. How six free-text answers become a countable theme. The last step is a threshold, not a statistic: three people is the point at which something stops being one person’s experience.
  • Database
  • App integration
  • Machine learning
  • Management
  • Analytics
  • Front-end & mobile

Three, not two

Two people mentioning scheduling in a quarter is a coincidence in a small business, and reporting it produces a manager who can name both of them. Three is the smallest number at which a pattern is more likely than a coincidence and at which attribution starts to get genuinely difficult.

The threshold is a setting, and the honest guidance is that raising it is safer than lowering it. A business with forty leavers a year could reasonably use five.

Unmatched answers

An answer that matches no theme is counted as unmatched rather than discarded, and a rising unmatched count is one of the more useful signals the system produces: it means people are leaving for reasons your theme list does not contain. Reviewing the unmatched answers — which requires somebody senior reading raw text, deliberately and rarely — is how the theme list gets updated.

The comparison nobody else has

Because the same person answers the same six questions twice, the system can do something no ordinary exit process can: compare them.

How the two rounds of answers differ, and what that revealsA horizontal row of five boxes. Ask one themes: a better offer, mentioned six times. Ask two themes: a better offer, mentioned twice. What replaced it: scheduling, four times. The gap: what people would not say on their last day. Reported: as a gap rather than a quote. A note says the gap is the whole argument for asking twice and it is measurable.WHAT CHANGES BETWEEN THE TWO ASKSAsk 1 themesbetter offer: 6Ask 2 themesbetter offer: 2What replaced itscheduling: 4The gapwhat people would not sayReportedas a gap, not a quoteThe gap is the whole argument for asking twice, and it is measurable.
Fig 2. The difference between the two rounds of answers. The shift from a better offer to something specific is the measurable form of the thing everybody suspects about exit interviews.
  • Management
  • Analytics
  • Front-end & mobile

That shift — six people citing a better offer on the way out, two of them still saying it six weeks later, and scheduling appearing in its place — is the single most persuasive output this system produces, because it is the measurable version of something everybody already suspects about exit interviews and nobody has ever been able to demonstrate.

Next: what the report says, and everything it refuses to say.

All posts