Software often starts with a simple idea of correctness. The system is right or it is wrong. You write tests to prove which one it is, and a bug is something to fix.
That idea works well up to a few million events a day. After that, it stops helping in the same way. At high volume, the guarantees you would like can become physically impossible, or they can cost more than the business they protect.
Telecoms is where many engineers meet this problem for the first time. The shift can feel like an invitation to accept sloppiness. The real task is more exacting. You have to be precise about which kind of wrong matters, where it matters, and how you will know.
Correctness changes when the volume changes
At a thousand records a day, you can check every record. You can reprocess anything suspect. A person can look at anomalies and decide what happened.
At a billion, the checking becomes a system larger than the system being checked. Reprocessing everything can mean a week of compute. Correctness then changes shape. You stop treating it as something you can assert for every individual record across the whole system. You start treating it as a statistical property of large populations, with much stronger guarantees kept for the small subset where one bad record really matters.
The common mistake is applying one standard everywhere. Some teams demand per-record perfection across the whole pipeline. Then the pipeline cannot keep up and starts dropping data, which is a worse kind of wrong. Other teams accept eventual approximate correctness everywhere. Then they discover that the same tolerance that works for an aggregate usage graph can be catastrophic for the record that creates somebody’s bill.
Group the data by the cost of being wrong
The first exercise on a high volume system is to divide event types by the consequence of getting one wrong. It is remarkable how rarely teams have done this explicitly.
Usually there are three bands, and they need completely different engineering.
- A single error is a financial or regulatory event: a charge, a settlement between operators, a lawful intercept record, or a regulatory submission.
- A single error is invisible, but a systematic error is serious: quality metrics, capacity planning inputs, or churn signals.
- Nobody would ever notice one missing record, and the real requirement is that the loss rate stays bounded and known.
The first band is usually a small fraction of total volume. It justifies per-record durability, exactly-once semantics, an audit trail and reconciliation.
The second band needs statistical guarantees and drift detection, because the danger is a pattern of wrong results rather than one bad record.
The third band can tolerate loss, as long as that loss is bounded and known.
If you engineer all three bands to the standard of the first, the system will fail to handle the load. If you engineer all three bands to the standard of the third, you can end up in front of a regulator.
Exactly-once delivery usually means exactly-once effects
Every distributed system conversation eventually reaches exactly-once delivery. Across a network partition, with independent failure domains, that guarantee is unavailable in the general case.
The useful version is at-least-once delivery combined with idempotent processing, where retrying the same event produces the same result instead of doing the work twice. That combination produces exactly-once effects, which is what the business needed in the first place.
The practical consequence is that idempotency has to be built in from the start. It is the property that makes the architecture legitimate. It belongs in the identity of every event at the point the event is created.
An event needs a key that stays stable across retries. That key has to come from the originating system rather than being assigned later in the pipeline. Processing has to see the key and recognise that it has already acted on it, cheaply and at volume.
That usually means keeping a bounded window rather than an infinite ledger. Choosing that window is a real decision. It has a real failure mode at the edges.
Time causes more trouble than throughput
Throughput is a solved problem in the sense that you can buy your way through it with partitioning and horizontal scale. Time is harder.
In any system where events are generated in many places and processed centrally, the order in which things happened and the order in which they arrive are different. The gap between those two orders is unbounded in theory and awkward in practice.
A record generated on a device at 23:58, buffered because the device was out of coverage, and delivered at 00:14, belongs to yesterday for billing and to today for arrival.
If the pipeline uses arrival time for everything, month-end is wrong every single month. The error is small enough to be argued about and large enough to matter.
So event time and processing time are carried separately. Watermarks, which mark when a time window is considered closed, decide when processing should move on. The system also needs an explicit policy for data that arrives after its window closed.
Discarding late data can be a legitimate choice. Accidentally discarding it because nobody made the decision is a different failure.
The system needs an explicit answer when a stage cannot keep up
When a downstream stage cannot keep up, something has to give. There are only three options.
- Buffer.
- Slow the producer.
- Discard.
Systems that have not made this choice in advance discover their answer during an incident. The answer is usually the worst one: data is silently discarded because a queue filled up and the default behaviour was to overwrite.
We make the choice explicit per data class, using the bands above. High consequence events get durable buffering with alerting when the buffer grows. The system slows rather than loses those events.
Low consequence events are allowed to shed. The shedding is counted and reported, so the loss rate is a number on a dashboard rather than a rumour.
The critical discipline is that shedding is never silent. A system quietly discarding two percent of events looks exactly like a system working perfectly, until somebody reconciles it against an external source months later.
Testing has to target the failures that scale creates
You cannot load test a billion event pipeline by running a billion events through a copy of production every time somebody changes a line.
What you can do is test the properties that hold regardless of scale. Then you verify scaling behaviour less often and deliberately.
Each kind of test finds a different class of defect.
- Property-based tests catch ordering and idempotency errors that unit tests miss, because they generate awkward interleavings a human would not think to write.
- Replay of a captured production hour against a new version, comparing outputs, catches regressions in aggregate behaviour.
- A periodic full-scale soak, run against a genuinely representative volume, catches resource exhaustion problems that only appear after nine hours.
Skipping any of those tests means shipping that class of defect to production.
An independent count keeps the pipeline honest
The single most valuable thing we build into high volume systems is an independent count.
The count has to come from the source and be compared against the destination, on a schedule. It also has to be produced by code that shares no libraries with the pipeline it is checking.
That independence matters enormously. A checker built from the same code, using the same assumptions, will confirm the same errors.
We have seen a system report perfect health for months while dropping a whole category of records. The health check and the pipeline both derived their view from the same upstream filter. An independent count would have caught it on day one, and did, once it existed.
Schema change is a slow emergency
At low volume, you can change a data structure by deploying a migration and moving on.
At high volume, with producers you do not control and consumers you have never met, a schema change becomes a multi-month coordination problem. If it goes wrong, it corrupts a category of data quietly rather than loudly.
The failure is rarely a crash. It is a field that silently starts arriving as a string where it used to be a number. Somewhere in the middle, it gets coerced. Then it produces plausible but wrong aggregates for eleven weeks before somebody notices the graph has the wrong shape.
The rules that work are unglamorous, and they need to be stated plainly.
- Producers may add fields and may never remove or repurpose them.
- Consumers must ignore fields they do not recognise rather than rejecting the record.
- Every event carries a schema version, and the version is validated at the boundary rather than trusted.
- Anything that fails validation goes to a quarantine stream with the reason attached.
- That quarantine stream has an owner and an alert.
A quarantine nobody reads is a delete with extra steps.
The expensive part is the discipline rather than the mechanism. It only takes one team, under deadline pressure, repurposing an existing field because adding a new one required a conversation, to poison a year of history. We put that rule in writing at the start of these engagements for exactly that reason.
High cardinality can take down monitoring
There is a specific and very common way that high volume systems fall over. The pipeline keeps running, and the observability stack built around it collapses.
Somebody adds a metric tagged with a customer identifier, a device identifier, or a URL path that contains one. At a thousand customers that is fine. At forty million devices it generates forty million distinct time series. The monitoring system either falls over or starts costing more than the infrastructure it observes.
We treat label cardinality, meaning the number of distinct values a metric label can take, as a design constraint with a hard budget. We treat it the same way we treat latency.
Identifiers belong in logs and traces, where you can sample aggressively and look one up when you need it. Metrics get bounded labels only.
- Region.
- Data class.
- Schema version.
- Outcome.
If somebody needs per-customer numbers, that belongs as a query against the data rather than as a gauge in the metrics system.
This sounds like an operations detail, and it is the difference between an observability bill of a few thousand a month and one that arrives as a crisis. It is also, in our experience, the single most common cause of a monitoring outage during an incident, precisely when you need it most.
The dashboard should show flow, delay and loss
The dashboard needs the signals that expose high volume failure modes.
- Ingest rate and processing rate, side by side, because the gap is the thing that matters.
- Buffer depth per stage, with the age of the oldest item rather than just the count.
- Shed count by data class, which should be zero for the high consequence band.
- Independent reconciliation difference, absolute rather than percentage.
- Late arrival distribution, so a change in device or network behaviour is visible before it distorts a month end.
CPU stays off this dashboard. It has never once been the thing that told us something was wrong first.
The standard is chosen, bounded and measured error
Accepting that some errors are tolerable can feel like a lowering of standards. Engineers resist that idea, correctly, because the same argument is used to justify genuine sloppiness.
The distinction is whether the tolerance is chosen, bounded, measured and reviewed, or whether it emerges from a system nobody understood.
A team that can tell you their loss rate, which data it applies to, why that is acceptable, and what would happen if it doubled, is operating to a far higher standard than a team that believes they lose nothing and has never checked. The second team is usually wrong, and finds out from a customer.
More on how we work in this sector on our telecommunications page, and on the related argument about ledgers in financial services.
