Conventional monitoring starts from a simple assumption: when something is wrong, something breaks. An exception is thrown. A status code changes. A queue backs up. The system announces its own failure, and you build alerts around that announcement.
Probabilistic systems behave differently. A model can produce a confidently wrong answer and still return a 200, in normal latency, with well-formed output. Every conventional signal says the system is healthy. That is the most dangerous property of this technology in production, and it is barely discussed.
So observability has to answer a different question. You need to know whether the system is still as right as it was.
Conventional monitoring still catches the plumbing
Conventional monitoring still matters. Keep all of it.
Error rates catch the plumbing: timeouts, rate limits, schema violations and provider outages. Latency, particularly at the tail, catches the inputs that trigger pathological generation. Throughput catches upstream changes in volume. Cost belongs here too. In this field it is a first-class production metric rather than a monthly report, because it catches runaway retries before they become a bill.
These signals are necessary, and they are the easy half. They will not tell you that the answers got worse.
You need signals that show whether the answers changed
The useful signals are the ones that show movement in behaviour. They tell you when the system is drifting away from the version someone last checked.
Refusal rate shows when the system stops answering
Refusal rate tells you how often the system declines rather than answering. It is the single most informative number in a probabilistic system, and almost nobody instruments it.
A rising refusal rate means the inputs have changed, the corpus has drifted, or the model has become more cautious after an update. A falling refusal rate is worse. It can mean genuine improvement, or it can mean the system has started guessing where it used to decline. The second case is invisible in every other metric.
We alert on movement in either direction, rather than on a fixed threshold.
Confidence distribution shows the shape of certainty
For confidence, the average is the wrong thing to watch. The shape matters. A healthy system produces a bimodal distribution: mostly confident, with a small tail of genuinely uncertain cases routed to a human.
When that distribution flattens or shifts toward the middle, something has changed about the inputs. That usually happens before anyone reports a quality problem. It is the earliest available warning, and it costs nothing to record.
Escalation and override rate show where humans disagree
If there is a human queue, you need to know how much reaches it and how often the human disagrees with the system’s proposal.
The override rate is the closest thing to ground truth available in production. People already doing the work generate it for free. It is the number most worth putting on a dashboard where a business owner will see it.
Scheduled evaluation catches silent model changes
Scheduled evaluation means running the held-out set, the examples kept aside for checking quality, against production on a schedule rather than only in CI, the automated test pipeline.
This is the signal that catches silent model changes. Your code did not deploy, your inputs look normal, and the pass rate dropped four points overnight because a provider updated something. Without a scheduled run, you learn this from a customer weeks later.
Log what the model was given as well as what it returned
Most teams log what the model returned. Far fewer log what it was given. The second record is what you need when investigating a bad answer.
For a retrieval system, log the query, the retrieved chunk identifiers, their scores, and which chunks made it into the context. When an answer is wrong, the first question is whether the right material was present. You cannot answer that later without the record.
Where the content is sensitive, store identifiers and hashes rather than full text, with a way to reconstruct under proper access control. You need reproducibility rather than a second copy of the corpus in your logging system.
Record every version that could explain a behaviour shift
Every inference should record four versions, all of which change independently and any of which can explain a behaviour shift:
- Model identifier and version.
- Prompt version.
- Retrieval configuration version.
- Code version.
Without this, an investigation into last Tuesday’s degradation cannot establish what was different about last Tuesday. With it, the answer is usually visible in one query.
This is cheap to add at the start and painful to retrofit, because the historical data lacks it exactly when you need the comparison.
Sample normal production output for human review
Automated signals catch drift. They do not catch a systematic error that the system is confidently and consistently making, because there is nothing to compare against.
The only defence is periodic human review of sampled production output. Review the ordinary, high-confidence, apparently-fine outputs. The escalated cases are already known to be hard.
Small numbers are enough. Twenty a week, reviewed by someone who knows the domain, finds systematic problems that ran undetected for months. Every client we have persuaded to do this has found something in the first month, which is either a strong argument for the practice or a depressing comment on the state of these systems.
Alert on movement rather than fixed thresholds
Conventional alerting is threshold-based: page when error rate exceeds one percent. That works when there is a correct value and deviations are failures.
Here there is often no correct value. A refusal rate of eight percent might be perfectly healthy for one system and alarming for another. What matters is that it was six percent last week and is eleven percent today.
So the alerts compare against a rolling baseline and fire on significant movement in either direction. It generates more alerts initially, and they are more informative, because every one of them corresponds to something actually changing.
Put cost on the same dashboard as health and quality
This advice is unusual for observability, and specific to this field.
Put spend per request, per feature and per customer on the same screen as latency and error rate, with an alert and a circuit breaker. Inference cost is the only production metric in ordinary software that can ruin a quarter overnight through a configuration error. It is routinely tracked in a finance system on a monthly cycle, where it is discovered far too late.
We have seen a retry misconfiguration produce a weekend’s spend larger than the preceding month. Nothing alerted, because nothing was failing.
A usable dashboard follows the order of an investigation
A usable dashboard contains eight panels, in this order, because the order encodes how an investigation actually proceeds.
- Requests and error rate, for the plumbing.
- Latency at p50, p95 and p99, because the tail is where the interesting inputs are.
- Refusal rate against its rolling baseline.
- Confidence distribution, as a histogram rather than an average.
- Escalation volume and human override rate.
- Scheduled evaluation pass rate over time, annotated with deploys and known model updates.
- Spend, per unit and total, with the alert threshold marked.
- Version composition of live traffic, so you can see when something moved.
An investigation almost always walks down that list. The panel where the line moves tells you which of the four independent variables changed.
Three incidents show why these panels matter
The case for each panel is easier to make with the incident it would have caught.
The provider update. A document classification system ran at 96 percent for four months. One morning it dropped to 91. No deploy, no input change, no errors. The scheduled evaluation caught it within a day. The version panel showed the provider had moved the default model. The fix was pinning to the previous version while the prompts were adjusted. Without a scheduled run, the first signal would have been a downstream team noticing misfiled documents weeks later.
The corpus drift. A support assistant began refusing more often, climbing from four percent to nine over ten days. Nothing was broken. A product launch had introduced a category of question the knowledge base did not cover, and the system was correctly declining. The refusal alert turned what would have been a slow trickle of user complaints into a content task with a clear scope.
The confident regression. An extraction system started returning the wrong date field on one supplier’s invoices after that supplier changed their template. Accuracy on the evaluation set was unaffected, because the set predated the change. Error rate, latency and refusal rate were all normal. It was found by the weekly human sample, three weeks in. That is the failure mode nothing automated catches, and the argument for the sample.
Build the first pieces in the order that gives the most return
Nobody builds all eight panels on day one. We recommend this order, by return on effort.
- Log inputs alongside outputs with version stamps. It is nearly free at the start and impossible to reconstruct later, and every investigation depends on it.
- Run the scheduled evaluation. It is the only automated signal that connects to correctness rather than to health.
- Track refusal rate with a rolling baseline. It is the cheapest early warning available.
- Add the spend alert with a hard stop. This is the one that protects the budget rather than the quality.
- Run the human sample. It costs an hour a week and finds what nothing else can.
Everything after that is refinement. Teams that do the first five have a system they can operate. Teams with an elaborate dashboard and no evaluation set have a system that looks monitored.
Different people need different views at different times
A dashboard nobody looks at is documentation. There are three audiences and three cadences, and they need different views.
- Engineers, continuously. Errors, latency, versions and spend, with paging alerts on the plumbing. Standard practice, no different from any other service.
- Whoever owns the outcome, weekly. Refusal rate, override rate and evaluation pass rate, in a single short summary rather than a dashboard they have to visit. This person usually is not technical and is the one who will notice that a nine percent refusal rate matters commercially because it maps to a customer segment.
- Whoever carries the risk, monthly. In regulated work this is a compliance or clinical or finance function. They want the trend, the incidents, the human sample findings and what changed. This is also the audience that will eventually be asked what the system did in a given month, so the report doubles as the evidence.
Setting those three up at the start is far easier than retrofitting them after someone asks an awkward question.
These signals show whether the system still behaves like the checked version
None of this tells you the system is right. It tells you whether it is behaving as it did when someone last checked that it was right.
That distinction matters, and it is where the discipline lives. The evaluation set is the only thing connecting production behaviour to correctness, and it is a snapshot of a judgement someone made at a point in time. If the set was wrong, or has drifted from what users actually need, every metric downstream is measuring fidelity to the wrong target with great precision.
That is why the periodic human review is not optional. It is the only step in the whole apparatus that can notice the evaluation set itself has stopped being right.
More on building the set in the piece on evaluation harnesses, and on how this fits our delivery standard in how we work.
