The first thing we ask for on a manufacturing engagement is permission to change nothing.
No writes. No control. No new hardware on the control network. We ask for a one way tap on the telemetry that already exists, and three weeks to watch.
Clients find this frustrating, and we understand why. They have a problem, they have a budget, and the supplier’s first proposal is to sit quietly and observe. It can sound like billing for a survey.
That quiet start is the single most valuable thing we do on these projects. The reason is simple. The plant has been running for years. It works. The fastest way to destroy a relationship with the people who run it is to break something in month one.
A bad deployment costs the line, not a rollback
In most software, a bad deployment costs you a rollback and an apology. On a production line it costs you the line.
A stoppage at a mid-sized plant runs into thousands per hour before you count scrap. If the stoppage happens because the new system wrote a value somewhere unexpected, the cost goes beyond the money. You have just proven to every operator in the building that the software cannot be trusted, and you will not get a second hearing.
That imbalance should govern the entire delivery approach. Often, it does not. Teams arrive with the habits of web development, where shipping fast and iterating is correct because the cost of being wrong is low.
Those habits are actively dangerous here. The plant knows it, even if the project sponsor does not.
We start by observing for three weeks
Weeks one to three are phase zero. We tap the existing telemetry through a one way boundary, typically a data diode or a strictly outbound gateway, and we build a picture.
Nothing we deploy can write to anything. Nothing we deploy is on the control network. If our entire system fell over, the plant would not notice.
The exit condition is a working agreement with the plant’s own view of reality. A document alone is not enough. Our numbers have to match what the plant already believes.
If we think a machine ran for eleven hours yesterday and the shift supervisor knows it ran for nine and a half, we are wrong. We do not proceed until we understand why.
Usually the disagreement comes down to definitions, such as:
- whether changeover counts as running
- whether a micro-stop under two minutes is recorded
- whether the shift boundary is at the hour or at the handover
Those definitions are where the interesting disagreements live. Finding them before you build anything is worth a great deal.
The first useful product is visibility
Weeks four to eight are phase one. We build dashboards for supervisors showing real production state rather than planned state.
There are still no writes. There is still nothing on the control network.
The exit condition for this phase is behavioural rather than technical, and we insist on it. The plant uses our screen instead of the whiteboard, unprompted, for two consecutive weeks.
That means they use it because it is more useful. Meeting requests do not count.
This is a hard bar, and we have failed it. On one engagement the screen was accurate, fast and ignored. It showed the metrics head office cared about rather than the three numbers the shift supervisor actually used to make decisions.
We rebuilt it around those three numbers and it was adopted in four days. The lesson was not about dashboards. We had asked the wrong people what mattered.
The model advises before it controls anything
Weeks nine to fourteen are phase two. Now models enter the picture, and they raise recommendations to a person who accepts or rejects them.
The system still cannot write anything. A maintenance recommendation appears in a queue, a human decides, and the decision is recorded along with the outcome.
That record matters for three reasons at the same time:
- the training data
- the evidence for the business case
- the mechanism by which the plant develops trust
The exit condition is an acceptance rate high enough that the plant relies on the queue. That rate is agreed in advance with the people who work it, rather than imposed on them.
We have had clients want to skip this phase and go straight to closed loop control, on the grounds that the model is accurate. Accuracy does not remove the constraint. Trust is the constraint, and trust is earned in exactly one way: by being right in public over a period of time while the stakes are low.
Writing back starts in week fifteen
Week fifteen onward is phase three. Only now does anything write back, only to systems agreed in writing, and always behind the existing safety interlocks rather than through them.
That last clause is non-negotiable, and the reason deserves to be explicit. The interlocks were designed by people who understood the physical process and its failure modes. They have been validated. In many cases they are part of a certification.
Software that bypasses them to be more efficient has replaced a well understood safety argument with a poorly understood one.
Our system proposes setpoints. The interlocks retain absolute authority to refuse them. If that costs a few percent of theoretical optimum, that is a price worth paying.
The exit condition here is a documented rollback that the plant has executed at least once as a drill, rather than one that exists on paper and has never been tried.
The plant thanked us because the line kept running
The phrase in the title is close to something an operations manager actually said to us. What he was thanking us for was not the software. It was that fourteen weeks had passed without anybody from IT breaking his line.
He had been through three previous projects. Each one arrived confident, deployed early, caused an incident, and left the plant with a system that people worked around because they did not trust it.
The phased approach was the first time a supplier had treated the line as the constraint rather than as an obstacle to the project plan.
The commercial conversation has to show delivery in each phase
Fourteen weeks before any control action is a difficult thing to sell against a competitor promising results in six. We are explicit about it in proposals, and we state what each phase produces so it reads as delivery rather than delay.
Each early phase produces something concrete:
- Phase zero produces an agreed data dictionary and a set of numbers the plant confirms.
- Phase one produces a working visibility layer that has value on its own and can be kept even if the project stops.
- Phase two produces a measured model with a documented acceptance rate.
Each phase is a deliverable a client can point at. Each also has an exit condition the plant agreed to, which means the project cannot quietly slide forward on optimism.
OT and IT work from different instincts
There is a cultural gap on these projects that no amount of technology will bridge, and pretending otherwise wastes months.
Operational technology teams optimise for availability and safety over a horizon measured in decades. IT teams optimise for change velocity and feature delivery. Both are correct within their own world, and their instincts are close to opposite.
The practical resolution we use is to accept the OT constraints as requirements rather than as negotiating positions.
Those constraints are specific:
- No new protocols on the control network.
- Read paths and write paths physically separated.
- Change windows that fit the maintenance schedule rather than the sprint.
Everything we deploy runs on the IT side of the boundary and reaches across it in one direction only, until phase three, which is agreed separately and specifically.
Security starts with the physical consequence
Threat modelling in this environment starts from what a failure would mean in the building. That comes before a generic checklist.
We care about data exfiltration, but the first question is whether any path exists by which a compromised component could cause a physical outcome. The answer must be structurally no rather than probably not.
That is why the one way boundary matters so much in the early phases. The boundary gives more than caution. For fourteen weeks there is no mechanism, compromised or otherwise, by which our system can affect the process.
That is a far stronger statement than any amount of access control, and it is available essentially for free at the start of a project.
The data is messier than the pitch deck says
Every proposal in this sector is written as though the telemetry is clean, timestamped, complete and semantically consistent. It never is.
Phase zero exists largely to find out how the data is broken before anyone has committed to a model that assumes otherwise.
The recurring problems are consistent across plants:
- Tags named by whoever installed the sensor, with three different conventions layered by three different contractors over fifteen years.
- Timestamps from PLCs whose clocks have drifted and were never disciplined to a time source.
- Values that are reported on change rather than on interval, so a flat signal is indistinguishable from a dead sensor.
- Units that differ between two machines doing the same job because they came from different vendors.
- A decommissioned line that still emits data because nobody unplugged it.
None of that is anybody’s fault. It is the accumulated reality of a working plant.
The only serious way to deal with it is to catalogue it early, agree what each tag actually means with someone who was there, and write that down as a deliverable.
Skipping it does not make the problem go away. It relocates it to the moment a model produces a confident recommendation based on a sensor that has been reading a constant value since 2019.
A model can be right for six weeks and then drift
Manufacturing processes drift with things that are not in the data.
Those sources of drift include:
- ambient temperature
- humidity
- a change of raw material supplier
- a tool approaching the end of its life
- a different operator on nights
A model trained on eight weeks of summer production can be excellent and then quietly degrade in November for reasons no one connects to the software.
This is why the advisory phase cannot be treated as a formality that can be compressed. It is the mechanism by which you observe a model across enough real variation to know whether it generalises.
It is also why we insist on monitoring the input distribution rather than only the output accuracy. By the time accuracy has visibly fallen, the recommendations have been wrong for a while, and somebody has already stopped trusting the queue.
The operational answer is unglamorous:
- Alert on drift in the inputs.
- Retrain on a schedule rather than when something breaks.
- Keep the held-out set updated with recent production so the benchmark does not slowly become historical.
A model that is measured against last year is measuring the wrong plant.
We measure behaviour, trust and safety boundaries
The measurements we use are the ones that show whether the plant can rely on the system:
- Agreement between our figures and the plant’s own, per definition, before anything else proceeds.
- Unprompted use of our screens, measured rather than assumed.
- Recommendation acceptance rate, and the reasons given for rejections.
- Number of times our system has affected the process. Zero until phase three, deliberately.
- Rollback executed as a drill, with a date.
The honest cost is time before control
This approach is slower to first control action and no cheaper. What it buys is that the system is still running three years later, and that the operations team defends it rather than tolerating it.
We have taken over two systems that were built the fast way. In both cases the technology was fine and the deployment was dead, because an early incident had convinced the people who work the line that the software was a liability.
Recovering from that is far more expensive than fourteen patient weeks, and on one of those projects we did not manage it at all.
More on how we work in this sector on our manufacturing page, and on the delivery method in how we work.
