How recovery is honestly measured
This category reports conversion rates that are wrong by a large factor, and the reason is simple enough that it is worth stating plainly: a substantial number of people who abandon a form come back and finish it without any prompting at all, and the naive measurement credits every one of them to the message.
Key takeaways
- Many abandoners return on their own. A naive rate credits the message with all of them.
- A holdout — a share who are deliberately not messaged — is the only honest measurement.
- Ten per cent is enough, and it costs a small number of recoveries to know the truth.
- An honest uplift is usually modest, and modest is still worth having.
- The complaint and opt-out rate belongs in the same report as the uplift.
The people who come back anyway
- App integration
- Management
- Analytics
- Front-end & mobile
Why a holdout rather than a before-and-after
The obvious alternative — compare the recovery rate before the system existed with after — is confounded by everything else that changed. Seasonality, a site redesign, a different mix of traffic. A holdout running concurrently is the only comparison where the two groups differ in one thing.
Ten per cent is enough at typical volumes and it costs a small number of recoveries a month. It is worth paying: without it, nobody can say whether the system is doing anything, and a system nobody can evaluate tends to acquire a second message and then a third.
Keeping the holdout honest
The holdout has to be assigned randomly and permanently at the moment of abandonment, not chosen afterwards, and the assignment has to be recorded before the outcome is known. Assigning it retrospectively, or excluding holdout cases that look unusual, produces a number that is worse than not measuring.
It also has to survive somebody wanting to switch it off. The pressure to message the holdout arrives about four months in, framed as “we know it works now”, and the correct answer is that knowing it still works next year requires it to keep running.
What an honest number looks like
- App integration
- Machine learning
- Security & identity
- Analytics
- Front-end & mobile
Eight extra enquiries a quarter is a genuinely modest result and it is worth stating as such rather than dressing it up. For a business where an enquiry is worth a few hundred pounds, eight is a good return on a small system; for one where an enquiry is worth twenty, it probably is not worth building at all, and knowing that is useful.
The drop from three hundred and ten abandonments to one hundred and four that pass the basis check is the other number worth looking at, and it is the system behaving correctly. Two thirds of abandonments are people who typed an address somewhere without context, and not contacting them is the point.
The number next to the uplift
The complaint and opt-out rate belongs in the same table, always, because the uplift on its own invites the obvious optimisation and the two numbers together do not.
Reported together, every quarter
- Uplift: 9 percentage points against the holdout.
- Opt-outs: 3 of 94 messaged asked not to be followed up again.
- Complaints: 0 spam reports, 1 reply asking how we got their address.
- That last one matters even at a count of one. It means somebody did not recognise the context, which is a signal about the form rather than about them.
- If opt-outs exceed about 5%, the basis check is too loose and should be tightened rather than the wording softened.
That last rule is the one worth writing down somewhere durable. When a recovery message generates complaints, the instinct is to rewrite the message, and the message is almost never the problem. The problem is that it was sent to somebody who did not have enough context to recognise why, which is a basis question.
Next: what all of this costs to run.
All posts