Skip to content
allanninal.dev/build
Collaborate

Part 3 of 7 · Invoice dispute triager series ~5 min read

How a dispute gets classified

Classification is the only genuinely model-shaped task in this system, and it is a narrow one: read a customer’s sentence and say which of six things they mean. The interesting design question is not how to make it accurate. It is what to do when it is not.

Key takeaways

  • Six classes, plus “cannot tell”, which is a normal and useful outcome.
  • A dispute can carry two reasons, and the message says so rather than picking one.
  • The model gets the invoice lines in the prompt, so a quantity claim can be checked for plausibility.
  • A low-confidence class gathers nothing and hands over the raw message.
  • Every classification is stored with the message, so accuracy can be measured rather than assumed.

One call, seven possible answers

How a dispute is classified into one of six reasonsA vertical chain of five steps entered by a box labelled Recognised dispute, carrying the invoice. Step one classifies with a single Bedrock call, grounded by the invoice lines included in the prompt. Step two asks whether the classification is confident enough against a floor you set; if not it exits to Hand over raw, with no evidence gathered. Step three asks whether one reason or two were claimed; two exits to Gather for both, saying so in the message. Step four asks whether the dispute is about the invoice or about the work; the work exits to the relationship owner rather than accounts. Step five hands one reason to the gatherer. A note says every exit is a real destination and none of them is a rejection.AWS ACCOUNTRecognised disputewith the invoiceClassifyone Bedrock callThe invoice linesin the promptConfident enough?the floor is yoursHand over rawno evidence gatherednoOne reason or two?Gather for bothsay so in the messagetwoAbout the invoice?or about the workTo the relationship ownernot to accountsthe workTo the gathererwith one reasonEvery exit is a real destination. None of them is a rejection.
Fig 1. Classification and its four outcomes. The escape hatches are as important as the happy path, because acting confidently on a wrong class wastes more time than not classifying.
  • App integration
  • Machine learning
  • Analytics
  • People

Why the invoice lines go in the prompt

“We only got 6, not 10” is ambiguous without the invoice. Six of what? If the invoice has one line for 10 pallets, it is a quantity dispute on that line. If it has four lines and one of them is for 10 boxes of something, it is a quantity dispute on that line specifically, and knowing which line matters enormously for what evidence gets pulled.

So the prompt includes the invoice lines, and the output includes which line is being disputed where it can be determined. That is not the model doing arithmetic — it is the model matching a customer’s phrasing to a line description, which is precisely what it is good at.

Two reasons at once

Disputes are frequently compound: “we only got 6, and the price is higher than the quote anyway”. The temptation is to pick the dominant one, and it is a mistake, because resolving the quantity issue and leaving the price issue produces a second dispute a week later that looks like the first one was handled badly.

So a dispute can carry up to two reasons, evidence is gathered for both, and the message says so plainly at the top: “two issues raised”. Beyond two, the system stops classifying and hands the message over raw, because a message with three distinct complaints in it is a conversation rather than a dispute.

The confidence floor

Below the floor, nothing is gathered and the raw message goes to a person with a note saying the system could not tell what was being claimed. That feels like a failure and is the correct behaviour, for a specific reason: gathering the wrong evidence is worse than gathering none.

Why an uncertain classification is handed over rather than guessedA horizontal row of five boxes. Right class: the evidence is attached and the decision is quick. No class: the raw message is handed over honestly. Wrong class: the wrong paperwork is attached. Cost of wrong: two round trips, because the person must first realise the evidence is irrelevant. Cost of none: one round trip. A note says a wrong class costs more than no class, so the floor is set high rather than low.WHY "I CANNOT TELL" BEATS A GUESSRight classevidence attachedNo classraw message, honestWrong classwrong paperworkCost of wrongtwo round tripsCost of noneone round tripA wrong class costs more than no class, so the floor is set high rather than low.
Fig 2. The asymmetry that sets the confidence floor. Being wrong costs a person time twice; being honest costs it once.
  • Machine learning
  • Management
  • People

Measuring it

Every classification is stored alongside the raw message and, once a person has resolved the dispute, alongside what the reason actually turned out to be. That is a labelled dataset that accumulates for free, and after a few months it answers a question most people guess at: how often is this thing right?

In practice the answer tends to be that quantity, duplicate and price classify very reliably, “never ordered” and “wrong entity” are frequently confused with each other, and “not about the invoice” is the one worth watching, because misrouting one of those into an accounts queue is how a service complaint sits unanswered for a week.

Next: what the gatherer actually fetches for each reason.

All posts