Every year the same conversation happens in September. Someone asks whether the site will hold up on Black Friday. Someone else says it held up last year. That exchange is treated as a risk assessment, and it is not one, because last year is not evidence about this year unless nothing changed, and something always changed.
The useful version of that conversation starts with a number. Not “will it hold up” but “what exactly are we expecting, per minute, at the worst moment, and what have we proven the system does at that number”. Everything else is a feeling.
Averages are the enemy
Retail traffic is not distributed. It is spiky, and the spikes are the entire commercial event. A campaign that does two hundred thousand orders across a day might do eight thousand of them in a single ten minute window when an email lands and a countdown expires simultaneously.
If you size against the day, you will be roughly sixty times under-provisioned for the moment that matters. If you size against the hour, you are still wrong, because the hour contains the ten minutes. The unit that matters is the busiest minute you can plausibly have, and the only way to know it is to look at your own history at minute granularity, which most analytics tools will not show you by default.
So the first thing we do on a peak engagement is not architectural. It is to go back through last year’s logs and find the actual busiest sixty seconds, then ask marketing what they intend to do differently this time. A larger list, a shorter window, or a single national television slot each multiply that number in ways that are entirely predictable in advance and entirely surprising on the day.
Agree the number, in writing
We ask for a figure and we write it into the scope: orders per minute at peak, concurrent sessions, and the catalogue size at the time. That number becomes the specification. Everything afterwards is measured against it, and if the number changes because the campaign got more ambitious, that is a change to the specification with a cost attached, not a surprise for the engineering team to absorb quietly.
This sounds bureaucratic. It is the opposite. It is the thing that lets everyone stop arguing from intuition. When somebody asks whether we can add a live personalisation widget to the basket page in week minus two, the answer is not a debate about taste. It is a measurement against a known ceiling.
Load test to failure, not to comfort
Most load testing stops at the target and reports success. That tells you almost nothing useful. If you test to your expected peak and it passes, you know one fact: you are above the line somewhere. You do not know by how much, and you do not know what happens when you cross it.
We test until something breaks, every time. The point is not the number at which it breaks, though that is useful. The point is to find out how it breaks, because that is the behaviour you will actually experience.
There are only a few outcomes and they are very different. The system might slow down evenly, which is survivable. It might queue, which is often fine if the queue is bounded and honest. It might start returning errors on a fraction of requests, which is bad but recoverable. Or it might fall over entirely and take several minutes to come back, during which the retry storm from every waiting browser guarantees it falls over again. That last one is the failure mode that turns a busy hour into a lost day, and you can only discover it by going past the edge on purpose, in an environment where it does not matter.
The first bottleneck is almost never the web servers
Web tiers scale horizontally and everybody knows it, so they are usually the part that has been thought about. In our experience the thing that actually gives way is one of these.
- A single database writer. Reads scale out easily. Writes go to one place, and at peak your writes are orders, which are the ones you cannot drop.
- Inventory decrement under contention. Thousands of requests wanting the same row for the same popular item, serialising behind a lock.
- A synchronous call to something outside your control. Tax calculation, address validation, fraud scoring, a loyalty service. Each one is a third party with its own capacity, and your checkout is only as fast as the slowest of them.
- Session storage. Cheap and invisible at normal load, a hard ceiling at ten times it.
- Log volume. A surprising number of outages are a disk filling up with the logs written by the incident.
None of these show up at four times normal traffic. They show up at forty, which is why the test has to go there.
Degrade on purpose, before you have to
The most valuable work in the six weeks before a campaign is usually not making things faster. It is deciding, in advance and in daylight, what you are willing to lose.
A storefront has a hierarchy whether or not anyone has written it down. Checkout is essential. Search is close. Recommendations, reviews, recently viewed, live stock counts on category pages, and the chat widget are not, however much someone likes them. If those are wired in such a way that any of them failing can slow or break the checkout, then you have accepted that a recommendation engine can cost you the day.
So we cut them apart deliberately. Non-essential services get short timeouts and a defined empty state, and a circuit breaker so that once they are struggling we stop calling them entirely rather than waiting each time. The page renders without them. Nobody buying a coat has ever abandoned the purchase because the “customers also viewed” strip was missing, and plenty have abandoned because the page took nine seconds.
Doing this in advance is straightforward engineering. Doing it during an incident, with someone senior standing behind you, is not.
Idempotent orders are not an edge case
At peak, a meaningful fraction of your customers are on poor connections, on phones, tapping a button that appears not to have responded. They will tap it again. That is not user error, it is the only rational thing to do when nothing happens.
If your checkout can turn two taps into two orders and two charges, you will find out at exactly the moment you have the most customers and the least support capacity. The fix is ordinary: a client-generated key on the submission, stored and honoured server side, returning the original outcome on a repeat rather than creating a second one. It costs a day to build and it is the difference between a quiet Monday and several hundred refunds.
Caching is not capacity, and the difference bites
There is a reasonable instinct that says most of this is solved by putting a cache in front of it. For the catalogue, that is broadly true and you should absolutely do it. For the parts that matter at peak, it is a trap.
The pages you can cache aggressively are the ones where every visitor sees the same thing. The moment a page shows stock, a price for a logged in customer, a basket count or a personalised block, it either becomes uncacheable or it becomes a correctness problem. Serving a cached “in stock” for four minutes on a line with eleven units left is how you sell forty of them.
Worse, a cache changes the shape of failure rather than removing it. Under normal load it absorbs almost everything and your origin looks idle, which is reassuring and misleading. At the start of a campaign, when a new promotion invalidates a swathe of keys at once, every one of those requests arrives at the origin simultaneously. The cache did not reduce your peak, it concentrated it. We test with a cold cache for exactly this reason, because the first minute of a campaign is a cold cache by definition.
The technique that helps is not more caching, it is request coalescing: when a thousand requests miss the same key at the same instant, one of them goes to the origin and the other nine hundred and ninety nine wait for that answer. It is a small piece of code and it turns a thundering herd into a single request.
Rehearse the incident
A week before, we run the failure with the actual people who will be on call. Not a document review. We break something in production-like conditions and watch what happens, including the human parts: who notices, how they are told, who decides to switch the feature flag, and how long all that takes.
The technical findings are usually minor. The process findings are not. The most common discovery is that the person with the authority to degrade a feature is not the person watching the dashboard, and that the gap between them is fifteen minutes of somebody trying a phone number. Fifteen minutes at peak is a lot of orders.
On the day, watch orders and not CPU
Infrastructure metrics tell you how the machines feel. They do not tell you whether you are selling. We put one number on the main screen: orders per minute, against the same minute last year and against the expected curve.
The reason is that every serious incident we have worked shows up in the business metric before it shows up in the technical ones, or shows up there only. A payment provider silently declining a card type does not move CPU at all. A change to a tax service that adds four seconds to checkout will show as slightly elevated latency and a cliff in conversion. If you are watching the machines, you will find that an hour later.
What good looks like in numbers
On the last peak programme we ran, the shape was this. Agreed target of eleven hundred orders per minute. Load tested to failure at week minus five, which turned out to be six hundred and forty. First bottleneck was inventory contention on a handful of promoted lines. Fixed with a reservation model rather than a direct decrement, which took nine days. Retested at week minus three: twenty one hundred, with the next ceiling being the payment provider’s own rate limit, which we negotiated upward.
On the day, actual peak was thirteen hundred and forty, comfortably above the target and comfortably below the proven ceiling. Nothing was degraded. Nobody was woken up. The headroom was one point nine times the number the business expected, which is roughly where we aim: enough that a pleasant surprise stays pleasant.
The work that made that possible happened in September and October, not on the day. That is the entire point. Capacity is not something you have. It is something you proved, in advance, against a number somebody was willing to write down.
More on how we approach this in ecommerce and retail, and on the engineering commitments behind it in how we work.
