Reason Codes, Not Confidence Scores

Most document-reading systems report a confidence score. A number between 0 and 1, or a percentage, attached to a field or a document, saying how sure the model is.

It looks like exactly the information a reviewer needs. It is almost never useful, and it is worth understanding why before you build a review process around one.

What a confidence score does not tell you

It does not tell you what is wrong. “72% confident” gives a reviewer no idea what to look at. Compare it to “the invoice claims 8 units; the posted receipt shows 5 uninvoiced; difference +3”. One of those tells you what to do next.

It is not comparable between documents. A model’s 0.9 on a crisp born-digital PDF and its 0.9 on a phone photograph of a fax do not mean the same thing. The number is not calibrated across inputs, so a threshold that works for one population misbehaves on another.

High confidence and wrong is a real state. The failure that costs money is not the model saying “I’m unsure”. It is the model reading a plausible-looking number, confidently, and being wrong. A confidence threshold catches the cases where the system already suspected something, and misses precisely the cases it did not.

It cannot be reported on. “We had 40 documents below 80% last month” tells you nothing about your vendors, your process or your data. It tells you something about the model’s self-assessment, which is not a business fact.

Nobody can pick the threshold. Set it at 95% and everything queues. Set it at 70% and things slip through. There is no principled way to choose, so it gets tuned until the queue is a bearable size — which means the threshold ends up encoding how much review capacity you have, not how much risk you are willing to take.

What a reviewer actually needs

Three things, in order:

1. What check failed — a stable, named code.
2. The arithmetic — what was expected, what was found, the difference, and the tolerance it was measured against.
3. What to do next — the route out.

None of those is a probability. All of them are facts.

An exception that reads ERR-QTY-OVER with a factor row stating expected 5, actual 8, delta +3, tolerance 0 is a complete diagnosis. The reviewer knows the receipt showed 5, the vendor asked for 8, and the next step is to re-count the receipt or ask for a credit.

Why a closed set of codes matters

Free-text explanations — including good ones written by a model — cannot be counted.

A fixed, documented set of codes can be:

• Filtered. Show me every open ERR-PRICE-VAR.
• Trended. ERR-UNKNOWN-ITM is up 40% this quarter — somebody’s item references need attention.
• Attributed. This vendor produces 60% of our ERR-QTY-OVER exceptions, which is a conversation to have with them.
• Automated against. Route documents with this code to that approver.

None of that is possible over prose that varies per document. A closed set converts your exception queue from a to-do list into a data set about your suppliers and your master data.

It also means the codes must be stable strings that never change and are never translated. A code that means one thing in one language and another elsewhere is not an identifier; it is a caption. Anything that reports on codes needs them to be literal and permanent.

Where a model’s narrative belongs

There is a legitimate place for plain-language explanation: as an optional rewording of a decision that has already been made, on top of the factors, changing nothing.

The rules that keep it honest are simple, and worth checking in any product that offers one:

• It cannot change a decision.
• It cannot add a fact that is not in the factor rows.
• It cannot introduce a number.

Under those constraints a narrative is a readability feature. Without them, it is the model’s opinion presented as the system’s finding — and a reviewer has no way to tell which.

The deeper point

A confidence score is a statement about the reader. A reason code is a statement about the document.

Accounts payable is a domain where the second is available. You have a purchase order. You have a posted receipt. You can say, as a matter of arithmetic, that the claim disagrees with them and by exactly how much.

When a definite answer is available, reporting an estimate of your own certainty instead is a strange thing to do. Confidence scores are what you fall back on when there is nothing to check against — and on a PO-backed invoice, there is.

What this implies for evaluating a product

Ask to see an exception on screen. Then ask three questions:

1. Is what I am looking at a percentage, or a named code with the arithmetic?
2. Are the codes a fixed set I could build a report on, or generated per document?
3. When two things are wrong at once, which one is shown, and is that rule something I can predict?

The answers separate a product that will let you manage your AP process from one that will give you a queue you can only work through one document at a time.

DocumentQ’s own set is documented in full in the reason code reference, including the ranking rule that decides which code is shown when several apply.