Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Subscription audit bot series ~6 min read

How charges become a subscription

This is the only genuinely interesting algorithm in the series, and it is still not very interesting — which is the point. Finding subscriptions in a transaction feed is a clustering problem that people reach for machine learning to solve and that is better solved by noticing that subscriptions are, by definition, charges that happen at regular intervals.

Key takeaways

  • Grouping is on interval regularity, not merchant name similarity.
  • Four patterns: monthly, annual, quarterly, and per-seat monthly with a drifting amount.
  • Annual subscriptions need two years of feed to see, and they are the expensive ones.
  • A pattern that breaks is a finding: a price rise, a cancellation, or a failed payment.
  • Two charges is a coincidence. Three at a regular interval is a subscription.

Group on interval, not on name

The instinct is to group by merchant: put all the charges from the same place together and see which repeat. It works badly, because merchant strings for the same service vary between charges — a processor prefix appears and disappears, a city is appended, a reference number is included. Meanwhile genuinely different things share a merchant, because one payment processor fronts hundreds of small services.

How repeated charges are grouped into a subscriptionA vertical chain of five steps entered by a box labelled All transactions, covering twenty-four months if available. Step one buckets by cleaned merchant name as a rough first pass. Step two subdivides within a bucket by amount, exactly first and then within fifteen per cent. Step three asks whether the intervals are regular, with gaps within a few days of each other; irregular charges exit to Not a subscription and are left alone. Step four asks whether there are three or more charges, since two is a coincidence; two exits to Watch it, as not enough yet. Step five declares a subscription with its interval, amount and next expected date. A note says everything here is arithmetic on dates and no model is involved in finding a subscription.AWS ACCOUNTAll transactions24 months if you have itBucket by cleaned merchanta rough first passWithin a bucket, by amountexact, then +/- 15%Intervals regular?gaps within a few daysNot a subscriptionleave it aloneirregularThree or more?two is a coincidenceWatch itnot enough yettwoA subscriptioninterval, amount,next expected dateEverything here is arithmetic on dates. No model is involved in finding a subscription.
Fig 1. How a set of charges becomes a subscription. Bucketing by merchant is only a first pass; the actual test is whether the gaps between charges are regular.
  • App integration
  • Security & identity
  • Management
  • Analytics

The four patterns

PatternLooks likeWhy it needs its own handling
MonthlySame amount, gaps of 28–31 daysThe easy case. Three charges is enough to be confident.
AnnualSame amount, gaps of 360–370 daysNeeds two years of feed to see three charges. Usually the largest amounts.
QuarterlySame amount, gaps of 88–95 daysEasily mistaken for irregular spending on a short feed.
Per-seat monthlyMonthly gaps, amount rising in stepsAmount changes as headcount changes, so exact-amount grouping misses it entirely.

Why annual is the hard and valuable case

Annual subscriptions are where the money is — they are typically five to ten times the value of a monthly one and are exactly the ones nobody remembers. They are also the hardest to detect, because seeing three charges takes two years of transaction history, and most businesses can only export twelve or eighteen months.

So annual gets a relaxed rule: two charges roughly a year apart, from the same merchant, for the same amount, is treated as a probable annual subscription rather than waiting for a third. It is flagged as probable in the question, which is honest and costs nothing — “this looks like an annual subscription, renewing about the 14th of next month” is a perfectly actionable sentence even when the system is only reasonably sure.

When a pattern breaks

A subscription that stops behaving is at least as interesting as one that starts, and there are three cases worth telling apart.

Four ways an established subscription changes, and the resulting questionA horizontal row of five boxes. Amount rose: a price increase. Charge missing: either a cancellation or a failed payment. Interval halved: the subscription moved from annual to monthly. New merchant: the provider renamed or moved processor. One question: sent with a description of what changed. A note says a missing charge is the ambiguous one, because a cancellation and a failed payment look identical from a transaction feed.FOUR WAYS A SUBSCRIPTION CHANGESAmount rosea price increaseCharge missingcancelled, or failedInterval halvedmoved to monthlyNew merchantprovider renamedOne questionwith what changedA missing charge is the ambiguous one: a cancellation and a failed payment look identical.
Fig 2. The four ways an established subscription stops behaving, and why the missing charge needs a question rather than an assumption.
  • A price rise is the most common and the most quietly expensive. Software prices rise annually and nobody notices a monthly charge moving from £14.40 to £17.20. Stated as “up 19%, £33.60 a year more”, it is a decision.
  • A missing charge is genuinely ambiguous. Somebody cancelled it, or the card expired and the payment failed. Those need completely different responses, and the feed cannot tell them apart, so it asks: “no charge from PROJTOOL this month — did we cancel, or has the payment failed?”
  • An interval change almost always means somebody switched from annual to monthly billing, which usually costs about twenty per cent more per year. Worth a sentence.
  • A new merchant string for the same amount and interval is a rename, and recognising it as the same subscription rather than a new one preserves the history and the owner.

Next: how a subscription gets an owner and a purpose, which is where the money actually gets saved.

All posts