Supply chain security matured painfully. We learned to pin versions, to keep a bill of materials, to scan dependencies, to verify signatures, and to have an answer to the question of what would happen if a package we rely on were compromised. Those controls exist because of incidents that cost real money.
Then generative AI arrived, and a great many organisations wired a third party into the middle of their business processes without applying any of it. Not through negligence, but because the thing does not look like a dependency. It looks like an API, and APIs have a well understood procurement path.
The properties that made the old controls work
Consider what pinning a dependency actually gives you. A specific version, whose contents are fixed, which you can inspect, whose behaviour will not change until you decide to change it, and which you could vendor into your own repository if the upstream disappeared.
A hosted model has none of those properties. The version you called last month may be silently updated. The behaviour changes without a changelog you can diff. You cannot inspect it. You cannot vendor it. And if it is withdrawn, there is no equivalent you can drop in, because behaviour is not specified anywhere except in the outputs you have observed.
This is not an argument against using them. It is an argument for recognising that a control you are used to relying on is not available, and putting something else in its place.
Silent behaviour change is the distinctive risk
The failure mode that matters most is not an outage. Outages are loud, and everybody has a plan for them. It is a model update that changes behaviour in a way that is fine on average and wrong for your specific case.
An extraction pipeline that has worked for eight months starts returning dates in a different format for a subset of documents. A classifier becomes slightly more cautious and starts routing four percent more cases to human review, which nobody notices until the queue has a backlog. A summariser begins including a caveat paragraph that your downstream parser was not expecting.
None of those raise an error. They produce plausible output that is wrong for your purposes, and the only way to detect them is to be measuring continuously against a known-correct set.
The evaluation set is the control
This is the substitute for pinning, and it is the single most important thing to build. A held-out set of a few hundred examples, keyed correctly by a person, representative of your real distribution including the awkward cases, run against the model on a schedule and on every change you make.
It gives you a number. When the number moves, something changed, and you find out on a Tuesday from a dashboard rather than in November from a customer. It also gives you the ability to evaluate an alternative provider honestly, which is the foundation of any exit plan worth the name.
Building it is unglamorous and it is the work. We insist on it before anything is built on top of a model, and clients occasionally resist because it delays the interesting part by a fortnight. The fortnight is the cheapest insurance in the project.
Pin what you can, and know what you cannot
Most providers now offer dated or versioned model identifiers with a deprecation window. Use them. Calling a generic alias is convenient and it means your production behaviour can change on somebody else’s release schedule.
What pinning does not give you is permanence. Versions are retired, and the notice period is measured in months rather than years. So the plan has to include a migration path that is exercised rather than assumed, and the evaluation set is what makes that migration a measurable exercise instead of a leap.
Data flow is the second control
Every call to a hosted model is data leaving your perimeter. That is obvious when stated and routinely forgotten in implementation, particularly once retrieval is involved, because the context window is assembled dynamically and nobody is watching what ends up in it.
We ask a specific question on every integration: for the worst case document in this corpus, what exactly is transmitted. Often the answer surprises the client, because a retrieval step is pulling in adjacent chunks that contain material nobody intended to send. The remedy is filtering and redaction before the boundary, applied by code, with tests, rather than by an instruction in a prompt.
Then there are the contractual questions, which are ordinary vendor management: retention period, whether inputs are used for training, where processing occurs geographically, what the sub-processor list looks like, and what notification you get when it changes.
Concentration risk is real and rarely priced
If three of your core processes depend on one provider, you have a single point of failure with commercial as well as technical dimensions. Pricing can change. Terms can change. Capacity can be rationed during a period of high demand. A provider can decide your use case is outside their acceptable use policy.
The mitigation is not necessarily to run two providers in parallel, which is expensive and doubles your integration surface. It is to keep the abstraction honest enough that switching is a project rather than a rewrite, and to have measured at least once what the alternative actually does on your evaluation set. An exit plan that has never been tested is a document, not a plan.
The bill of materials needs a new section
Your software bill of materials probably lists libraries. It should also list models, and for each one: the provider, the exact version identifier in production, what data reaches it, which business processes depend on it, what the fallback is, and when the fallback was last exercised.
That list is short and almost nobody has it. Producing it is an afternoon’s work and it tends to surface at least one integration nobody remembered, usually built during a prototype and quietly promoted to production.
Open weights change the trade, not the problem
Running a model yourself removes the silent update problem entirely, which is a genuine advantage. It replaces it with the ordinary burdens of operating infrastructure: patching, capacity, security of the serving stack, and a supply chain question about where the weights came from and whether they are what they claim to be.
For some workloads that trade is clearly correct, particularly where data cannot leave a boundary or where a stable, auditable version is a regulatory requirement. For others the operational cost is not worth it. The point is that it is a trade to be made deliberately, with the properties written down, rather than by default in either direction.
The layers underneath the provider
Most teams stop their analysis at the provider, and there is more underneath. A typical model-backed feature depends on an orchestration framework, a vector database, an embedding model, a document parser, a tokeniser, and often a handful of glue libraries that were young when they were adopted. Each of those is an ordinary dependency with the ordinary risks, and several of them are unusually young for something load-bearing.
The embedding model deserves particular attention, because it is the one people forget is versioned. If it changes, your existing vectors are no longer comparable to new ones, and the failure is silent: retrieval quietly gets worse, relevance drops, and the symptom is users saying the assistant has become less useful. Re-embedding a large corpus is a real project, and it needs to be planned before it is urgent rather than discovered when a provider deprecates something.
The document parsers are the other quiet risk. They handle untrusted input by definition, they are frequently thin wrappers around older native libraries with long histories of memory safety issues, and they run wherever you decided to run them. A malicious PDF is a much more traditional attack than prompt injection, and it is the one that gets skipped because everyone is looking at the model.
Cost is a security property here
This is an unusual one and it catches people. With a conventional dependency, a bug produces an error. With a metered model, a bug produces a bill. A retry loop that does not back off, a prompt that accidentally includes an entire document instead of a chunk, or an agent that recurses one level deeper than intended can all turn a routine defect into a five figure invoice over a weekend.
It also creates a denial of wallet exposure. If an endpoint that calls a model is reachable by an unauthenticated user, somebody can spend your money at their leisure, and rate limits framed in requests per minute do not necessarily bound spend when a single request can carry a very large context.
So we treat token budgets as a control rather than as an accounting concern. Hard caps per request, per user and per day. Alerts on rate of spend rather than only on total. Context length bounded explicitly in code instead of relying on a model to truncate. And any endpoint that calls a model sitting behind authentication unless there is a specific reason it cannot, with a much tighter budget when it cannot.
What we put in client documentation
- Every model dependency, with the pinned version and the provider’s deprecation policy.
- The evaluation set, where it lives, and the current score for each dependency.
- What data crosses the boundary, and the redaction applied before it does.
- The fallback for each dependency, and the date it was last exercised.
- Who is alerted when an evaluation score moves beyond a threshold, and what they do.
That is five bullet points and it is more than most organisations have for a component sitting in the middle of a revenue-generating process.
The framing that helps
Treat the provider as a supplier of a component you cannot inspect, whose behaviour may change without notice, and on whom a business process depends. Stated that way, every control follows naturally, and none of them are novel. It is the same discipline applied to library dependencies, adapted for the fact that the usual mechanism for freezing behaviour is not available.
The organisations that get burned are not the ones that use these tools aggressively. They are the ones that integrated a probabilistic third party using the procurement process designed for a productivity application, and then found out what had changed when a customer told them.
More on this in our security work, and on how we build model-backed systems in AI development.
