We have seen the same pattern on enough engagements to treat it as a rule. When a retrieval system gives bad answers, the team has usually spent weeks changing the prompt and the model. They have spent almost no time checking what the model was given to read.
That is understandable. Prompting is fast to iterate and feels like progress. Retrieval is unglamorous. It involves reading documents, making decisions about them, and looking at improvements that mainly show up in an evaluation number.
But answer quality has a hard ceiling. The relevant passage has to be in the context in the first place. If the index did not surface it, no amount of model capability can recover from that.
Bad answers usually start before the model writes anything
When a system answers badly, the visible artefact is the answer. So that is what people debug first. Somebody rewrites the instructions, adds an example, tries a larger model, and sees a small improvement on the cases they are looking at.
The first check should be simpler. Did the system retrieve the correct passage?
That single diagnostic separates two different problems. In one case, the model saw the right thing and reasoned poorly. In the other, the model never saw the material it needed. The second case is far more common, and it needs a completely different fix.
We now measure retrieval separately before touching generation. On more than half the systems we have reviewed, the correct passage was absent from the context in a third or more of failing cases. No prompt fixes that.
Split documents around their own structure
The default approach is to split documents every eight hundred characters with an overlap. It is fast, universal and wrong for most real corpora.
Fixed-size chunking cuts through the middle of the thing you are trying to retrieve. The chunk that ends up in the index is a fragment. On its own, it is meaningless. Worse, it embeds to a vector, a numerical representation used for similarity search, that looks plausibly similar to a great many other fragments.
The damage is easy to miss until you look at the documents themselves. Fixed-size chunking can break the useful unit apart in several ways:
- A table is severed from its header.
- A clause is separated from the definition it depends on.
- A procedure loses its numbered step.
What works better is chunking on the document’s own structure, which means understanding the documents. The right unit depends on the content:
- Contracts split on clause boundaries with the definitions section attached to every chunk.
- Technical documentation splits on headings with the parent heading path prepended.
- Tables stay whole or get serialised row by row with the header repeated.
That is content work, and it is the highest-return hour available in most of these projects.
Specialist vocabulary needs exact matching
Semantic search is strong at conceptual similarity and weak at the exact tokens that matter most in specialist domains.
Those exact tokens are often the terms a user searches for. They include:
- Part numbers.
- Case references.
- Statutory citations.
- Drug names.
- Error codes.
These are precisely where embedding similarity is weakest. The model has no reason to place two adjacent part numbers far apart in vector space, and every reason to place them close.
The result is a system that answers general questions impressively and fails on the specific lookup that was the actual use case. Users notice this quickly. It destroys confidence faster than a wrong answer to a vague question.
Hybrid search covers exact terms and paraphrases
The fix is straightforward. Run keyword search and vector search together, combine the results, and let each cover the other’s weakness.
Keyword matching handles exact terms: identifiers, names, codes, quoted phrases. Vector search handles paraphrase, where the user asked about cancellation and the document says termination.
Fuse the two ranked lists and you get a candidate set that neither method produces alone.
In our experience hybrid beats pure vector search on real corpora almost every time. The gap is largest exactly where the business value is: specific, factual, high-stakes lookups.
Reranking earns its cost on larger, messier corpora
Reranking means retrieving broadly, then using a second model to score each candidate against the query and reorder the results. It works. It also adds latency and cost per query.
Our rule of thumb is simple. Reranking is worth it when the corpus is large enough that first-stage retrieval returns many plausible-but-wrong candidates. Small, clean corpora often have a first stage that is already precise.
That sounds obvious and is routinely ignored in both directions. Teams add a reranker to a two-hundred-document corpus. Other teams omit one from a corpus of four million.
Measure it. Run the evaluation with and without, compare the numbers against the latency and cost difference, and make an explicit decision rather than following a pattern from a blog post.
Metadata removes wrong candidates before ranking
Chunks carry text and, usually, nothing else. Adding structured metadata and filtering on it before ranking is close to free and frequently the largest single improvement available.
The metadata can describe the document and the circumstances where it applies. The useful fields include:
- Document date, so a query about current policy does not surface the 2019 version.
- Document type, so a query about a contract does not return a marketing page.
- Department.
- Region.
- Product line.
- Effective date range.
- Access permissions, which matter enormously and are covered below.
Filtering the candidate set before ranking is cheaper than ranking a larger set. It also gives better results, because it removes wrong answers rather than hoping they rank low.
Permissions must be checked before ranking
The most serious defect we find in enterprise retrieval systems is a security issue as well as a quality issue.
A system indexes everything a service account can read, then answers questions for any user. The user asks something innocuous and receives a synthesised answer drawing on a document they have no right to see.
Nothing was breached in the conventional sense. The retrieval layer simply had no concept of who was asking.
The correct design filters by the requesting user’s permissions at query time, before ranking, using the same authorisation source as the underlying system. Filtering afterwards fails, because the answer may already have been synthesised from material that was then removed.
This is the first thing we check on any inherited system, and it is wrong more often than not. We wrote about the broader version of this problem in the piece on scoped mandates for agents.
Measure retrieval before generation
You cannot improve what you do not measure separately. Build a retrieval evaluation before a generation evaluation.
The evaluation starts with real questions. For each one, have someone identify which passages contain the answer. Then measure whether your system returns them in the top k, at whatever k you actually pass to the model.
That number is your ceiling. If the correct passage appears in the top five only seventy percent of the time, then thirty percent of your answers are being generated without the necessary information. The model will confidently produce something anyway.
Every hour spent on prompting before fixing that is wasted.
The policy assistant failed because it retrieved the wrong documents
A client had built an internal assistant over about forty thousand policy and procedure documents. Staff used it for a fortnight and then stopped. The complaint was that it made things up.
It was answering from whatever it had been given, which was frequently the wrong document.
We built a retrieval evaluation from 120 real questions taken from the support queue. A person marked which document actually contained each answer. Baseline recall at k equals five was 58 percent. In other words, in more than four cases in ten the model never saw the correct source and produced a plausible answer from adjacent material.
We made three changes, in order:
- Chunking moved from 800-character windows to section boundaries with the heading path prepended, which took recall to 74 percent.
- Hybrid search added alongside vector similarity took it to 86 percent, with almost all of the gain on questions containing a policy reference number.
- Metadata filtering on document status, so superseded versions were excluded unless explicitly requested, took it to 93 percent.
Only then did we look at the prompt. The prompt work was minor. It mostly instructed the model to cite which document each claim came from and to say when the retrieved material did not answer the question.
Both behaviours are only possible once retrieval is good enough that saying so is rare rather than constant.
Total effort on retrieval was about nine days. The six weeks of prompt iteration before we arrived had moved the number by roughly four points.
Refusing weak answers made the system usable
One detail from that project is worth separating out, because it applies everywhere.
Once retrieval is measured, you can set a threshold below which the system declines to answer rather than synthesising from weak material. That single behaviour did more for user trust than the accuracy improvement did.
Staff had stopped using the original system because they could not tell a confident correct answer from a confident wrong one. Every answer required verification, and the tool saved nobody any time.
A system that says the policy library does not appear to cover this is immediately useful, because the answers it does give can be acted on.
The threshold is a business decision rather than a technical one. Set it with whoever owns the risk, measure how often it fires, and show the number to them monthly.
Start with retrieval before changing the prompt
The order matters more than any individual technique.
Work through the sequence before you tune the answer-writing part of the system:
- Build the retrieval evaluation set.
- Measure the baseline.
- Fix chunking against the actual document structure.
- Add hybrid search.
- Add metadata filtering.
- Re-measure.
- Only then look at generation, because only then is the model reliably being handed the right material.
On a recent engagement that sequence moved answer accuracy from around sixty percent to the low nineties without a single change to the prompt or the model.
The team had spent six weeks on prompting before we arrived. That reflects how this work is usually taught, rather than a failure by the team.
Retrieval only fits tasks built around finding passages
Retrieval has become the default answer to every question, so this needs to be said plainly.
There are several cases where retrieval is the wrong architecture:
- A corpus small enough to fit in context should be put in context.
- A question that is really a database query with a natural language wrapper should be handled by writing the query.
- An answer that requires reasoning across the entire corpus rather than locating a passage in it needs a different approach entirely, usually a precomputed summary layer.
We have talked clients out of retrieval systems several times, and the reason is always one of those three. It is a cheaper conversation than building one and discovering it afterwards.
More on how we structure this kind of work on AI development, and on why the evaluation comes first in how we work.
