The same question comes up on every healthcare engagement. Can we send this to a model?
The answer is almost never a simple yes or no. HIPAA does not contain the word model. It was written for a world of faxes and filing cabinets. That does not make the rules unusable. HIPAA contains a framework, and that framework maps cleanly onto model use once we stop looking for AI-specific guidance and apply the rules that already exist.
We work through the issue by asking what data leaves the clinical environment, who receives it, what agreement governs that work, and what we can prove afterwards.
The first question is who receives the data and under what agreement
The useful question is who becomes a business associate, what did they receive, and can we demonstrate afterwards what happened.
A model provider processing protected health information on your behalf is a business associate, meaning a vendor that handles PHI for you as part of a service. That reading follows directly from the definition. Once that is true, the whole apparatus applies: a signed business associate agreement, defined permitted uses, breach notification obligations, and the right to audit.
So the first practical question is whether the provider will sign a BAA. Several will. Several will not, or will only for specific enterprise tiers. That single fact removes most of the ambiguity before any technical discussion begins.
We put every data flow into one of four categories
We sort every data flow into one of four buckets. The bucket determines the controls.
- Fully identified PHI. Names, record numbers, dates of service, everything. This goes only to a provider under a BAA, in a defined region, with zero retention configured and documented. It is logged in full.
- Limited data set. Direct identifiers are removed, but dates and some geography remain. That is often what makes clinical analysis possible at all. It requires a data use agreement. The risk is lower, but it is still regulated and still logged.
- De-identified under Safe Harbor. Eighteen specified identifier categories are removed. The data is no longer PHI, so the regulatory constraint lifts. Safe Harbor is stricter than people expect, and de-identification is harder than it looks.
- No patient data at all. Policy documents, clinical guidelines, drug references, internal procedures. This is frequently the highest-value application and it carries almost none of the constraint, which is why we look here first on every engagement.
That last category deserves attention. A very large share of what clinicians want is faster access to guidelines and protocols. That involves no patient data whatsoever. Teams often jump to the hard problem when the easy one has most of the value.
De-identification takes more than removing names
Safe Harbor lists eighteen categories. They include names, geography smaller than a state, all dates more precise than a year, phone numbers, email addresses, record numbers, device identifiers, biometrics, full-face photographs and more.
Three difficulties come up again and again.
- Free text hides everything. Structured fields are straightforward. Clinical notes contain names in running prose, references to the patient’s employer, the name of their village, and the date of their daughter’s wedding. Automated de-identification of clinical narrative is hard, and no tool is perfect.
- Dates carry clinical meaning. Removing precision below the year destroys the information a longitudinal analysis needs. This is why limited data sets exist. It is also why the honest answer to many requests is that you need a data use agreement rather than de-identification.
- Rare combinations re-identify. Remove every listed identifier and a sufficiently unusual combination of diagnosis, age band and region can still point to one person. Safe Harbor leaves that issue outside the required analysis. Expert determination includes it, and it is the more defensible route for anything being published or shared broadly.
The strongest uses are specific and reviewable
Specificity matters, because vague enthusiasm is what gets projects rejected at review. These are the places where AI earns its place.
- Documentation. This is the largest and least controversial area. A model turns a consultation into a structured note that the clinician edits and signs. The clinician remains the author, the model is a drafting tool, and the accountability chain is unchanged.
- Retrieval over guidelines and protocols. This uses no patient data. It answers the question a clinician would otherwise spend fifteen minutes finding.
- Triage support. This ranks a queue by clinical urgency rather than arrival order, with the reasoning shown to the person who signs it off.
- Coding and billing. The task is well specified, the volume is high and the current manual error rate is already known and non-trivial.
- Retrospective review. This identifies cases a human reviewer should look at again. It is underused and unusually safe, because nothing acts on the output except a person deciding where to spend attention.
We stop when no named person accepts the clinical decision
We stop at autonomous diagnosis. We also stop at anything that changes a clinical record without a named person accepting it.
Capability alone does not solve this. No regulator will accept a system whose reasoning cannot be reconstructed, and nobody in the organisation can carry the liability for an unreviewed clinical decision. That is an accountability structure, and it is correct.
We have declined work on this basis. The conversation is shorter at the proposal stage than at the incident stage.
The audit trail has to show what happened
The audit trail distinguishes a system that passes review from one that fails. It has to be designed in from the first schema, rather than added when someone asks.
Every inference involving patient data records the same core facts.
- Who requested it.
- Which patient’s data was involved.
- What exactly was sent.
- Which model and version received it.
- What came back.
- Whether a human accepted or overrode it.
- When it happened.
Those records are retained for the same period as the clinical record itself, because that is the period over which they may be questioned.
The specific thing auditors ask, and the thing most systems cannot answer, is what the model saw. They are asking for the actual assembled input for that specific request, reconstructible months later. A prompt template is not enough. If you have logged only the template and the patient identifier, you cannot reproduce the request once the underlying record has changed.
Six controls make approval much easier
Six things come up in every security review. Build them before being asked.
- Zero retention, evidenced. Do not rely on the provider’s marketing page. Document the contractual term and the configuration that enforces it.
- Region pinning. Processing happens in a named jurisdiction, with a technical control preventing failover elsewhere.
- Minimum necessary. Send only the fields the task requires. This is an existing HIPAA principle and it maps directly onto prompt construction.
- Named human accountability. Every output that reaches a clinical context has a recorded person who accepted it.
- Version pinning. The model cannot change under you without a documented review, because a behaviour change in a clinical tool is a change to a validated system.
- A tested rollback. Turning it off is a drill that has been executed, rather than a paragraph in a document.
Two similar requests can have completely different rules
The distinction between categories is easier to see when applied to real requests.
Request one: summarise this patient’s history before their appointment
This is identified PHI, unavoidably. The record is the point. So the provider needs to be under a BAA. The region is pinned. Zero retention is covered contractually and technically.
Minimum necessary applies, so the prompt carries the specific fields the summary needs rather than the whole chart. The output is a draft the clinician reads and accepts, and the acceptance is recorded. The audit entry is full, including the assembled input. This is buildable and it takes a security review to get there.
Request two: what does our protocol say about anticoagulation before this procedure?
This looks similar and is completely different. The protocol is an institutional document. No patient data is required to answer it. So there is no BAA question, no de-identification, no retention concern, and the audit requirement is ordinary rather than clinical.
Same clinician, same afternoon, two requests that feel alike. One is a regulated data flow and one is a document search. Sorting requests into those two piles early is most of the work. Doing it late is how projects acquire six months of review.
Security reviews usually ask about control, not the model
Having been through a number of these, we see the same questions. They are rarely about the model.
- Which vendor processes the data, under what agreement, and may we see it?
- What is the retention period at the vendor, and how is zero retention enforced rather than promised?
- In which jurisdiction does processing occur, and what prevents failover to another?
- Who at your organisation can see patient data, and how is that access revoked?
- Show us the audit record for a specific request from three months ago.
- What happens if the vendor has a breach? When are we told, and by whom?
- How do we turn this off, and has that been tested?
Every one of those is answerable in a sentence if the system was designed with them in mind. Each one can require weeks of retrofitting if it was missed. The sixth question is the one that most often stalls a project, because breach notification obligations flow through to the covered entity and nobody has read the vendor’s terms carefully enough to answer.
The common mistake is changing the data after approval
A pilot runs on de-identified or synthetic data, works beautifully, and gets approved on that basis. Production then quietly uses real records because the de-identification step was slow or lossy and someone made a pragmatic decision.
The system is now processing PHI under an approval granted for something else. Nothing looks different, no alarm fires, and the finding arrives at the next audit.
The defence is a technical control rather than a policy. The production path should be incapable of sending identified data to a provider that lacks a BAA. That must be enforced in code at the boundary, with the check logged. A rule in a document gives you a policy. A rule that throws an exception gives you a control.
Ask first whether patient data is needed
Before any of this, ask whether the work needs patient data at all.
On more than half the healthcare requests we receive, the honest answer is no. The clinician wants faster access to a protocol, or a draft of a letter, or a summary of a guideline. Reframing the problem to avoid PHI entirely removes the BAA, the de-identification, most of the audit burden and most of the review time.
It is the least technical intervention available and consistently the most valuable one. We ask it first on every engagement, before anyone opens a diagram.
More on how we build in this sector on the healthcare page, and our data handling commitments across all client work are on the trust centre.
