"Our AI is 95% accurate" is the most common sentence in AI marketing, and it says almost nothing. Accurate on what? Measured when, on which cases, by whom, against what? A number without those answers asks you to trust it. We would rather you could check it.
This is the method we use, and the rules we hold ourselves to.
What is the ground truth?
What really happened. For a quoting assistant, it is every enquiry the team already answered, next to the quotation they actually sent. The tool is scored on whether it would have suggested what an experienced person chose.
That ground truth is versioned. When a case turns out to be mislabelled, it is corrected in a new version, and every number says which version it was measured on.
What is a round?
A round is one set of changes, measured once. Every past case is run through the tool before and after the change, and the results are saved to a file. A number that has no saved run behind it does not count, however good it looks.
The comparison is paired: the same cases, before and after. That way a change that fixes twenty lines and breaks eight shows up as both, instead of as a net twelve that hides the eight.
Why counts instead of percentages?
Because a count carries its own denominator. "1,507 of 3,483 answered items" tells you how many cases there were; "43%" invites you to forget. We also state which round a number comes from and when it was measured, so a newer, better number can never be mixed up with an older one.
The count we quote on our home page is from round 12 of a hydraulics supplier's matcher: the first suggestion was the item staff actually quoted on 1,507 of 3,483 answered mailbox items. That is under half. It is on the page because every item behind it can be opened.
Who reads the misses?
People do. In the early rounds, every miss was read by hand and given a cause: a word misread, an item missing from the catalogue, a past quotation that was itself a mistake. Those causes decide the next round's changes, and the ones that turn out not to work are written down too, with the reason, so nobody tries them again.
Does the client see it?
Yes. The client's CRM has a Performance tab showing which cases got better and which got worse in each round, with a link to each case, so they can disagree with the grader. The evaluation imports the live algorithm and carries no copy of it, so an improvement measured is an improvement in the tool the team is using.
What do we refuse to publish?
Any figure without a run behind it. Savings estimates, "up to" numbers, and percentages in headlines. If a number is worth putting on a page, it is worth saying what it is out of, when it was measured, and where the file is.