There is a question we ask early on every engagement involving a model, and it often creates a pause. Take the most sensitive document in the corpus you are indexing. Show me, byte for byte, what leaves your infrastructure when a user asks a question that happens to retrieve it.
The pause happens because the final context sent to the model is assembled at runtime. Code written across several sprints chooses and combines it. Nobody has usually looked at the assembled result for an awkward case.
The team knows what they meant to send. The useful question is what is actually sent.
The risky part is the text your system adds at runtime
A direct call to a model is fairly easy to reason about. You have a prompt, and you can read what is in it. Retrieval augmented generation is different. The system chooses extra text to include at runtime, based on a similarity calculation.
That extra text is where the surprises tend to live. Several things routinely end up in the context which nobody intended:
- Adjacent chunks pulled in for continuity, which may cross a document boundary into something with a different sensitivity.
- Metadata attached during indexing, including file paths that reveal organisational structure or client names.
- Headers and footers containing distribution restrictions.
- In one case we found, an entire document’s contents because the chunking had failed on an unusual format and fallen back to including the whole file.
You cannot see that by reading only the code that constructs the prompt. The code builds a template. The template is filled later, when the query runs.
You need to inspect the assembled context
We build a way to dump the fully assembled context for a given query. Then we run the awkward cases through it before launch.
We do not use a sample of typical queries for this check. We use deliberately chosen adversarial ones: the query most likely to retrieve the personnel file, the query that would surface the unredacted contract, and the query a curious employee would try.
This takes an afternoon, and it has found something on every project where we have done it. Once the issue is visible, the fix is usually straightforward. Reading code almost never finds it.
Sensitive data should be removed before it crosses the boundary
The common mistake is to handle sensitive data with an instruction. Do not reveal personal information. Do not quote salary figures.
That is the same category error as trying to solve injection with a prompt. It asks a probabilistic system to enforce a policy, and it fails silently when it fails.
Redaction has to happen in deterministic code, before the data crosses the boundary, with tests. If national insurance numbers must not leave, then a function removes them and a test asserts that it does, using real-shaped examples.
If a document class must never be indexed, that is enforced at ingestion by a rule. A filter at query time can be bypassed by somebody.
The distinction matters when somebody asks for evidence. “We instruct the model not to” is not a control. “This function removes it, here is the test, here is the coverage” is.
Permissions need to limit retrieval before the model sees anything
The most consequential architectural decision in a retrieval system is where access control happens. The tempting answer is to index everything into one collection and filter the results by the requesting user afterwards.
That approach performs well and is simple to build. It also means the retrieval step has already read documents the user is not entitled to.
That is a problem even when the filter works correctly. The ranking has been influenced by documents the user cannot see. A single bug in the filter exposes everything rather than something.
It is a much worse problem when the model has already been handed the retrieved text before the filter runs. We have found that mistake in production more than once.
The pattern that holds up is to make permissions part of the retrieval query rather than a post-processing step. The index physically cannot return what the user may not read.
That usually means storing an access identifier alongside each chunk and filtering within the vector search rather than after it. It is more work at ingestion, particularly when permissions are inherited from a folder structure that changes. It converts an exposure that depends on application correctness into one that depends on the index refusing.
Permissions change after the index is built
The second half of the permissions problem is time. Somebody leaves a project and their access is revoked in the source system. The index was built last month.
Unless something propagates that change, the vector store still believes they may read it. It will keep believing that until the next full reindex.
The failure is quiet. It is exactly the kind of thing that surfaces during an audit rather than during testing.
We treat permission changes as events that invalidate index entries, the same way a document edit does. We also reconcile the index against the source of truth on a schedule, so drift is bounded and measured.
It costs a nightly job. The alternative is a system whose access control is accurate as of whenever somebody last thought about it.
Once data crosses the boundary, the contract matters
Once data has crossed, the questions are contractual. They are worth reading rather than assuming.
These are the questions we check:
- How long are inputs retained.
- Are they used for training, by default or on an opt-in basis.
- Does that answer differ between the consumer product and the API.
- Who at the provider can access them, and under what circumstances.
- Where geographically does processing occur, and does that change under load.
Providers vary substantially, and their terms change. We record the answers in client documentation with the date they were checked. A statement about a provider’s retention policy that was true in 2025 is not evidence about 2026.
Your own logs may expose more than the provider does
This is the finding that surprises clients most. The provider’s retention is governed by a contract with a large company that has an incentive to comply.
Your own logs are governed by whatever the team configured. They frequently contain the full prompt, which contains the sensitive material. They are then shipped to an aggregation service that a broad group can search.
We have seen prompts containing patient information in a log aggregator accessible to the whole engineering department, on a system whose provider contract had been negotiated carefully by the legal team. The perimeter was strong, and the leak was inside it.
The remedy is ordinary data hygiene applied to a new place:
- Log identifiers rather than content.
- Redact before writing.
- Set retention to the shortest useful period.
- Control access to the aggregator the way you would control access to the database.
Embeddings should be treated as sensitive data
A recurring misconception is that a vector is safe because it is a list of numbers. Embedding inversion research has repeatedly demonstrated that a meaningful amount of the original text can be reconstructed from its embedding, particularly for short texts.
In plain terms, a vector database containing embeddings of sensitive documents should be treated as containing the documents. It needs the same access controls, same encryption, same retention, and same answer when somebody asks where the data is.
If it lives in a hosted vector service, that service is a subprocessor holding sensitive data and belongs on the list.
The subprocessor list often grows after the contract is signed
By the time a typical model-backed feature is in production, the list of third parties touching the data is longer than the one negotiated at contract signature.
The list can include:
- The model provider.
- Possibly a separate embedding provider.
- A vector database, if hosted.
- An observability platform for traces, which by design captures inputs and outputs.
- An error tracker.
- Sometimes an evaluation service.
Each was added by an engineer solving a problem. Each is reasonable. Together, they represent a data flow nobody has mapped.
We produce that map as a deliverable. It is frequently the first time anybody has seen the whole picture on one page.
Deletion has to reach every derived copy
If a customer exercises a deletion right, what happens to the copy of their data in your vector index, your trace logs, your evaluation set, and your provider’s retention window?
The honest answer for most systems is that deletion covers the primary database and misses the rest. That is a compliance gap, and it is much easier to design for than to retrofit.
We keep a reference from every derived artefact back to the source record, so a deletion can cascade. We accept the storage cost of that mapping because the alternative is being unable to answer the question.
A region setting does not answer the whole residency question
Selecting an EU endpoint in a console feels like it settles the geography question. For the main inference call, it usually does. It rarely covers everything else in the path.
The rest of the path can include several places where data or access may sit outside that setting:
- Abuse monitoring may be performed centrally.
- Support access during an incident may come from another jurisdiction.
- A fallback capacity pool may sit elsewhere and be used silently under load.
- Trace data from your observability vendor may be stored on a different continent from the model calls it describes.
For most commercial work, none of that matters. For regulated data, it matters a great deal. The answers are obtainable if you ask specifically rather than accepting a region label at face value.
We ask three questions in writing:
- Where is inference performed.
- Where is any retained copy stored.
- Which jurisdictions can staff access it from.
The third is the one that is almost never volunteered. It is frequently the one a regulator cares about.
The evidence we produce for clients
For this work, we produce a concrete set of deliverables:
- A data flow diagram showing every boundary crossing, with the provider and the retention term at each.
- A dump of the assembled context for a set of deliberately awkward queries, reviewed by someone from the client’s side.
- The redaction functions, with tests, listed as a deliverable rather than as an implementation detail.
- The subprocessor list, dated, with a note on how it is kept current.
- A deletion trace showing where a single record’s derivatives live.
The question to hold yourself to
The useful question is not which provider you use, because everybody can answer that. The useful question is this: for the worst document in our corpus, what exactly left, who at the provider could see it, how long will they keep it, where else did we copy it, and can we delete all of that if asked.
An organisation that can answer that in a meeting has done the work. One that has to go and find out has an exposure whose size is currently unknown. That is a different and less comfortable position than having decided the exposure is acceptable.
More on how we approach this in cybersecurity services, and on building model-backed systems in AI development.
