Every year, the same conversation starts in September. Someone asks whether the site will hold up on Black Friday. Someone else says it held up last year.
That sounds reassuring, but it gives the business very little to work with. Last year only helps if nothing changed, and something always changed. The campaign changes. The list changes. The catalogue changes. The way customers arrive changes.
The useful conversation starts with a number. The question becomes: what exactly are we expecting, per minute, at the worst moment, and what have we proven the system does at that number. Without that number, everyone is working from a feeling.
Start with the busiest minute, because the average hides the risk
Retail traffic is spiky. The spikes are the commercial event. A campaign can do two hundred thousand orders across a day and still do eight thousand of them in a single ten minute window, because an email lands and a countdown expires simultaneously.
If you size against the day, you will be roughly sixty times under-provisioned for the moment that matters. Sizing against the hour still misses the risk, because the hour contains the ten minutes. The useful unit is the busiest minute you can plausibly have.
That minute usually sits inside your own history. You need to look at last year’s traffic at minute granularity, meaning the numbers broken down minute by minute. Most analytics tools will not show that view by default.
So the first thing we do on a peak engagement is not architectural work. We go back through last year’s logs and find the actual busiest sixty seconds. Then we ask marketing what they intend to do differently this time.
A larger list, a shorter window, or a single national television slot each multiply that number. Those changes are entirely predictable in advance and entirely surprising on the day when nobody has put them into the plan.
Write the target into the scope
We ask for a figure and we write it into the scope. That figure covers orders per minute at peak, concurrent sessions, and the catalogue size at the time.
That number becomes the specification. Everything after that is measured against it. If the campaign gets more ambitious and the number changes, the specification has changed too. That carries a cost, instead of becoming a surprise for the engineering team to absorb quietly.
This can sound bureaucratic. It does the opposite job. It lets everyone stop arguing from intuition.
When somebody asks whether we can add a live personalisation widget to the basket page in week minus two, the answer does not have to become a debate about taste. We measure that request against a known ceiling.
Test past the target so you know how failure behaves
Most load testing stops at the target and reports success. That tells you very little. If you test to your expected peak and it passes, you know one fact: the system is above the line somewhere.
You still do not know by how much. You also do not know what happens when the system crosses the line.
We test until something breaks, every time. The break number matters, but the more useful result is how the system breaks, because that is the behaviour you will experience during the campaign.
There are only a few outcomes, and they feel very different to the business:
- The system might slow down evenly, which is survivable.
- It might queue, which is often fine if the queue is bounded and honest.
- It might start returning errors on a fraction of requests, which is bad but recoverable.
- It might fall over entirely and take several minutes to come back.
The last failure is the one that turns a busy hour into a lost day. Every waiting browser retries, and that retry storm guarantees the system falls over again. You can only discover that pattern by going past the edge on purpose, in an environment where it does not matter.
The first limit is usually somewhere other than the web servers
Web tiers scale horizontally, meaning you can usually add more web servers to share the work. Everybody knows this, so the web tier is usually the part that has already been thought about.
In our experience, the part that gives way is usually one of these:
- A single database writer. Reads scale out easily. Writes go to one place, and at peak your writes are orders, which are the ones you cannot drop.
- Inventory decrement under contention. Thousands of requests want the same row for the same popular item, serialising behind a lock.
- A synchronous call to something outside your control. Tax calculation, address validation, fraud scoring, a loyalty service. Each one is a third party with its own capacity, and your checkout is only as fast as the slowest of them.
- Session storage. Cheap and invisible at normal load, a hard ceiling at ten times it.
- Log volume. A surprising number of outages are a disk filling up with the logs written by the incident.
These tend to stay hidden at four times normal traffic and appear at forty. That is why the test has to go there.
Decide in advance what the storefront can lose
The most valuable work in the six weeks before a campaign is usually deciding what you are willing to lose. That decision should happen in advance and in daylight.
A storefront has a hierarchy, whether anyone has written it down or not. Checkout is essential. Search is close. Other parts matter less during the peak moment, however much someone likes them.
The parts that can be cut are already familiar:
- Recommendations.
- Reviews.
- Recently viewed.
- Live stock counts on category pages.
- The chat widget.
If those parts are wired so that any of them failing can slow or break the checkout, then you have accepted that a recommendation engine can cost you the day.
So we cut them apart deliberately. Non-essential services get short timeouts and a defined empty state. They also get a circuit breaker, which stops calling a struggling service entirely rather than waiting each time. The page renders without them.
Nobody buying a coat has ever abandoned the purchase because the "customers also viewed" strip was missing. Plenty have abandoned because the page took nine seconds.
Doing this in advance is straightforward engineering. Doing it during an incident, with someone senior standing behind you, is a different situation.
Make repeated checkout taps safe
At peak, a meaningful fraction of your customers are on poor connections. Many are on phones, tapping a button that appears to have done nothing. They will tap it again. That is the rational thing to do when nothing happens.
If your checkout can turn two taps into two orders and two charges, you will find out at exactly the worst moment. You will have the most customers and the least support capacity.
The fix is ordinary. The checkout submission carries a client-generated key, stored and honoured server side. When the same key appears again, the server returns the original outcome instead of creating a second order.
It costs a day to build and it is the difference between a quiet Monday and several hundred refunds.
Use caching for the catalogue, and know where it stops helping
There is a reasonable instinct that says most of this is solved by putting a cache in front of it. For the catalogue, that is broadly true and you should absolutely do it.
For the parts that matter at peak, caching can become a trap. The pages you can cache aggressively are the ones where every visitor sees the same thing.
The moment a page shows stock, a price for a logged in customer, a basket count or a personalised block, one of two things happens. The page becomes uncacheable, or it becomes a correctness problem. Serving a cached "in stock" message for four minutes on a line with eleven units left is how you sell forty of them.
A cache also changes the shape of failure rather than removing it. Under normal load it absorbs almost everything and your origin looks idle. That can feel reassuring, while hiding the real peak.
At the start of a campaign, a new promotion can invalidate a swathe of keys at once. Every one of those requests then arrives at the origin simultaneously. The cache did not reduce your peak. It concentrated it.
We test with a cold cache for exactly this reason. The first minute of a campaign is a cold cache by definition.
The technique that helps is request coalescing. When a thousand requests miss the same key at the same instant, one of them goes to the origin and the other nine hundred and ninety nine wait for that answer. It is a small piece of code and it turns a thundering herd into a single request.
Rehearse the incident with the people who will be on call
A week before the campaign, we run the failure with the actual people who will be on call. This is a real exercise, not a document review.
We break something in production-like conditions and watch what happens. That includes the human parts: who notices, how they are told, who decides to switch the feature flag, and how long all that takes.
The technical findings are usually minor. The process findings matter more. The most common discovery is that the person with the authority to degrade a feature is not the person watching the dashboard.
The gap between them is fifteen minutes of somebody trying a phone number. Fifteen minutes at peak is a lot of orders.
On the day, watch orders before you watch CPU
Infrastructure metrics tell you how the machines feel. They do not tell you whether you are selling.
We put one number on the main screen: orders per minute, against the same minute last year and against the expected curve.
The reason is simple. Every serious incident we have worked shows up in the business metric before it shows up in the technical ones, or shows up there only.
A payment provider silently declining a card type does not move CPU at all. A change to a tax service that adds four seconds to checkout will show as slightly elevated latency and a cliff in conversion. If you are watching the machines, you will find that an hour later.
Good peak work leaves you with proven headroom
On the last peak programme we ran, the shape was this. The agreed target was eleven hundred orders per minute. We load tested to failure at week minus five, and the failure point turned out to be six hundred and forty.
The first bottleneck was inventory contention on a handful of promoted lines. We fixed it with a reservation model rather than a direct decrement, which took nine days.
We retested at week minus three. The result was twenty one hundred, with the next ceiling being the payment provider’s own rate limit, which we negotiated upward.
On the day, actual peak was thirteen hundred and forty. That was comfortably above the target and comfortably below the proven ceiling. Nothing was degraded. Nobody was woken up.
The headroom was one point nine times the number the business expected, which is roughly where we aim: enough that a pleasant surprise stays pleasant.
The work that made that possible happened in September and October, rather than on the day. That is the full argument. Capacity is something you proved, in advance, against a number somebody was willing to write down.
More on how we approach this in ecommerce and retail, and on the engineering commitments behind it in how we work.








