In conventional software, done is usually easy to answer. The tests pass. The behaviour matches the specification because the specification was executable all along.
Probabilistic systems change that. These are systems where the same input can produce different output. The output can also be wrong in ways that look grammatical, plausible and internally consistent. Nothing crashes. No exception is thrown. No assertion fails. The pipeline stays green.
That means correctness becomes a distribution rather than a boolean. You are no longer asking whether one fixed answer appeared. You are measuring how often the system gives an acceptable answer across many cases. The old machinery that told you whether to ship stops working.
What replaces it is the evaluation harness. The argument of this piece is simple: the harness is the specification itself. Until it exists, you do not have a product. You have an intention.
The harness is where the product is defined
Stripped down, an evaluation harness has three parts:
- A set of inputs.
- A definition of what a good output looks like for each.
- A scoring function that turns the comparison into a number you can track over time.
The first part is easy. The third is engineering. The second is where projects either succeed or quietly fail, because it forces someone to write down what correct means.
That is the same work as writing a specification. It is harder to avoid, because a vague specification can be nodded at in a meeting. A vague evaluation cannot be executed.
A prose specification leaves too much undecided
The usual order is specify, build, test. In this field that order fails, and it took us a few projects to work out why.
A prose specification will contain phrases like “accurately summarise”, “appropriate tone”, and “correctly identify the key terms”. Everyone agrees with those phrases. Nobody has agreed on anything concrete. Two competent people can read that document, build different systems, and both believe they met the spec.
Try to turn those same phrases into a score and the ambiguity becomes impossible to ignore. Accurately summarise against what reference? Produced by whom? Scored how, given that two valid summaries can share almost no words?
So the harness is where you discover that the specification was never finished. Building the harness does the hard work the prose document avoided.
Different jobs need different scoring methods
There are four main ways to score outputs. Each is honest in a different kind of task.
Exact match
Exact match means the output must equal the reference. It applies to extraction, classification, and anything with a canonical answer.
It is cheap, unambiguous, and the right default whenever the task admits it. Teams reach past it too quickly because it feels unsophisticated.
Structural match
Structural match checks the shape of the answer, with values allowed within tolerance. That might mean a date within a day, a total within a penny, or a required field present and typed correctly.
This covers most document work, and it is where the majority of commercial value sits.
Rubric scoring by a model
Rubric scoring means a second model judges the output against written criteria. It is useful for prose, and dangerous.
The judge has its own biases. It favours output that resembles its own style, and it will happily award high marks to fluent nonsense.
Use it only if you calibrate it. Have humans score a sample, compare the results, and measure the agreement. If a model judge and your domain expert agree less than about four times in five, the judge is measuring something other than what you care about.
Human scoring
Human scoring is expensive, slow, and the ground truth everything else is calibrated against. You cannot run it on every deploy.
You must run it periodically, or your cheaper measures drift away from reality without telling you.
The awkward cases matter most
A harness built only from representative examples measures the wrong thing. Representative cases are the ones the system will get right. The value is in the awkward tail.
Every set we build has a deliberately hostile section. That section includes cases like these:
- The ambiguous document.
- The one with two plausible answers.
- The one in the wrong language.
- The empty one.
- The one where the correct response is to refuse.
That last category matters most. A system that never refuses is a system that guesses, and a set with no refusal cases cannot detect it.
A useful heuristic is this: if your pass rate is above 97 percent on the first run, the set is too easy. It is telling you nothing you will not learn painfully later.
You need one set to develop against and one set to trust
Keep two sets. Keep a development set you may inspect, debug against, and argue about. Keep a sealed set you run rarely and never open.
Without the sealed set, contamination is inevitable and gradual. A production failure arrives. Someone adds the case. The fix makes it pass, and the number goes up.
Repeat that for six months and you have a system beautifully tuned to a set of cases that already happened. You also have an accuracy figure nobody should trust.
When the two diverge, the sealed one is right. We have had that conversation with clients more than once, and it is never a comfortable meeting.
An invoice harness shows how concrete this gets
To make this concrete, consider a task we have done several times: extracting structured data from supplier invoices.
The build happens as a sequence over a week.
- Day one, collect. Pull 800 real invoices from the last year, sampled to include every supplier rather than the top ten. This step alone surfaces things nobody mentioned: two suppliers who changed layout mid-year, one who sends scans of printouts, and a batch where the currency symbol is missing entirely.
- Day two, key them. Two people independently type the eleven target fields for each invoice. Independently matters. Where they disagree, a third person adjudicates and the disagreement is recorded. Those disagreements are the most valuable output of the whole week, because they are precisely the cases where the specification was ambiguous and nobody had noticed.
- Day three, argue about scoring. Is a date correct if it is right but formatted differently? Is a supplier name correct if it omits “Ltd”? Is a total correct if it is the gross rather than the net, when the invoice labels neither clearly? Every one of these is a business decision, and every one has to be settled before a number means anything.
- Day four, build the runner. Build something that takes a system, runs the set, produces per-field accuracy broken down by supplier, and writes the failures somewhere readable. Per-supplier breakdown is required: an aggregate of 96 percent can hide one supplier at 60 percent, and that supplier is somebody’s largest account.
- Day five, seal half of it. Split into a development set people may inspect and a sealed set that runs on a schedule and is never debugged against.
At the end of that week, nothing has been built. What exists is a definition of done that two parties have signed, an honest baseline, and a mechanism that will tell you the truth for the life of the system.
The first failures show where the spec is weak
The failure list from the first run is the most useful document in the project, and it is rarely about the model.
On the invoice job, the first run failed most often on documents where the relevant number appeared twice: once in a summary box and once in a line item. The model picked the wrong one about a fifth of the time. No amount of prompting fixed it reliably, because the ambiguity was genuinely in the document.
The fix was a rule agreed with the client about which position wins. That rule was added to the specification, applied consistently, and then measured.
That is a business decision surfaced by a test. That is what a specification is supposed to do.
The harness makes pricing and remedies possible
This is where the harness changes how the business works, as well as how the engineering works.
A measured pass rate on an agreed set is what makes a fixed price possible. Without it, a supplier quoting a fixed price for probabilistic work is gambling, and the gamble is priced into your quote whether or not anyone says so.
It is also what makes a remedy publishable. You cannot promise a refund on a failure you cannot define. It also lets a buyer compare two suppliers on something other than a demonstration, which today is almost the only basis on which these decisions get made.
This is the mechanism behind selling AI as defined jobs rather than as access. Every job on BotUp is backed by a harness, because the alternative is a published price with nothing behind it.
The first useful harness takes about a week
Being honest about the bill: for a job of moderate complexity, building the first useful harness is roughly a week of combined effort.
That week includes a domain expert producing and checking reference outputs, an engineer building the runner and scoring, and a real argument about what the scoring should be.
Clients push back on that week regularly. It is the week that decides whether the thing can be priced, measured, sold, and defended, so we hold it.
On engagements where we have been talked out of it, we have regretted it every time and usually rebuilt it later at greater cost.
The ongoing cost is smaller but real. The set needs maintenance as inputs drift, and the sealed set needs replenishing as cases leak into development.
The harness also needs to run after release
Most teams treat evaluation as a gate: run it in CI, meaning the automated build and test pipeline, and block the merge if it drops. That is necessary and insufficient.
The inputs your system sees change without any deploy on your side. Suppliers alter their document layouts. Customers start asking a new kind of question. The model behind your API endpoint can change under you with no announcement.
So the harness runs on a schedule against samples of live traffic, as well as against the fixed set on merge. Drift shows up as a slow decline that no deploy explains. Finding it in a dashboard is enormously cheaper than finding it in a customer complaint.
A shared number changes team arguments
There is a cultural effect worth naming. Once a number exists, arguments about quality change character.
Before the harness, quality discussions are aesthetic. Someone thinks the output feels wrong, someone else disagrees, and the argument is settled by seniority.
After the harness, the question is whether the pass rate moved. If the pass rate is fine while a person is unhappy, that is information. Either the harness is missing a criterion that matters, or the objection is a preference rather than a defect.
Both are useful findings, and both require the number.
It also removes the worst dynamic in this kind of work, which is the demo. Systems that impress in a live demonstration and disappoint in use are systems selected by anecdote. A harness makes anecdote inadmissible.
The one-line rule
If you cannot score it, you have not specified it. If you have not specified it, you cannot price it, sell it, defend it, or improve it deliberately. Everything else in a production AI system is downstream of that number existing.
We build the harness first on every engagement now, before the happy path works, because the alternative is discovering at the end that nobody ever agreed what the thing was supposed to do. More on where this sits in our process on how we work, and if you have a task you think is measurable, we will tell you honestly whether it is.
