Every payments product looks the same when the network behaves. Money goes in, a webhook arrives, a row appears, a balance updates. You can build that in a fortnight and it will pass a demo comfortably. The difference between that and a system a finance director will sign off is entirely in what happens on the days the network does not behave, and those days are not rare. They are Tuesday.
The feature that decides it is reconciliation. Not the report labelled reconciliation in your admin panel, which is usually a sum over the rows you already trust. The actual capability: given a period, prove that what your system believes happened matches what the outside world believes happened, and when it does not, say precisely where and why.
Why the demo is misleading
A payment is not an event. It is a conversation between several parties who each keep their own book, and who each have their own opinion about when something became true. Your system, the processor, the acquiring bank, the customer’s issuer, and your accounting package will all eventually agree. They will not agree at the same moment, and the gaps between them are where the money appears to go missing.
Most of the failures we are called in to fix are not failures of correctness in the ordinary sense. Nobody computed a total wrong. What happened is that two systems disagreed about the state of a transaction for eleven minutes, something in between read one of them, and a decision got made on a state that was true at the time and false afterwards. The code was right. The model of time was wrong.
The five conditions that break naive systems
We introduce these deliberately in staging before anything goes live, because production will introduce them without asking.
- The same webhook, twice. Providers retry. A callback you already processed arrives again, sometimes hours later, sometimes out of order with a newer one. The correct outcome is one ledger entry and an acknowledged duplicate, not two charges and a support ticket.
- The write succeeded, the confirmation never arrived. Your database committed and then the connection dropped before your service knew. On recovery the system has to determine which of those two worlds it is in, without a human reading logs.
- Two services disagree about now. Ordering by wall clock across machines is a bug waiting for a leap second or a slow NTP sync. If your sequence depends on comparing timestamps from different hosts, it will eventually be wrong and you will not be able to tell when it was.
- The batch half lands. Four hundred of a thousand movements clear and the rest fail. The reconciliation has to identify the six hundred precisely, not report a mismatched total and leave somebody to find them.
- A day is replayed at volume. Reprocessing a historic day must produce the identical ledger. If it does not, the ledger was never the source of truth, it was a cache of one.
Every one of those is testable. None of them is exotic. What makes them expensive is finding them after go live, when the evidence has aged and the person who can explain the anomaly has left.
Idempotency is a design, not a decorator
The common half-solution is to sprinkle idempotency keys on inbound endpoints and call it done. That handles the duplicate webhook and nothing else. Real idempotency is a property of the whole path: the key has to be derived from something the originator controls, it has to survive a retry from a different process, and the stored result has to be returned rather than recomputed, because recomputing may produce a different answer if anything upstream has moved.
The subtle part is what you store against the key. If you store “processed”, you have protected yourself against double writing but you cannot answer what the first attempt actually did. We store the full outcome, including the identifiers of everything created, so a repeat request returns the original answer byte for byte. That turns a retry from a risk into a non-event, which is exactly what the client integrating with you needs it to be.
Append only, and why totals should be derived
If there is one structural decision that separates systems that reconcile easily from systems that do not, it is this. Balances should not be stored and updated. They should be derived from an immutable sequence of movements.
The objection is performance, and it is a reasonable objection until you measure it. A running balance can be materialised, cached, and recomputed on demand, and the cost of doing so is trivial next to the cost of a stored balance that has drifted for reasons nobody can reconstruct. A stored balance is a claim. A derived balance is a proof, and when somebody asks why an account shows what it shows, you can produce the working.
It also makes correction honest. In a mutable model, fixing a mistake means editing history, and the fix is indistinguishable from the error afterwards. In an append only model, a correction is another movement with its own reason and author. Nothing is erased. The auditor sees what went wrong and what was done about it, which is a far better position to be in than a tidy ledger with no explanation.
What we actually build
The shape is consistent across the payment systems we have delivered.
- An event log of movements, immutable, with the originating reference from every external party attached to each one.
- A projection layer producing balances and statements, rebuildable from scratch at any time, and rebuilt nightly as a test that it still can be.
- A reconciliation job that pulls each counterparty’s own record for the period and compares it line by line, not in aggregate.
- An exceptions queue for anything that does not match, with an age and an owner, because a mismatch nobody is looking at is worse than no reconciliation at all.
- A dashboard whose primary number is the count of unexplained differences, which should be zero, and whose secondary number is the age of the oldest one.
That last point is the one clients push back on most often. They want a percentage. Ninety nine point nine percent matched sounds excellent. It is not, because the missing tenth of a percent is not distributed evenly across harmless small transactions. It clusters, and it clusters on exactly the cases that are unusual, which are the cases most likely to be somebody’s large and angry problem.
A worked example, with the figures rounded
A client processing around forty thousand transactions a month came to us with a recurring problem. Roughly every fortnight, the finance team found a difference between the platform and the processor’s own statement. It was never large, usually a few hundred pounds against a monthly volume in the millions, and it always resolved itself within a week or two. They had learned to live with it.
That phrase, learned to live with it, is the one that gets our attention. A difference that resolves itself is a difference nobody understands, and a mechanism you do not understand is not safe just because it has been small so far.
What was happening: refunds issued near midnight were being recorded against the day the request was made, while the processor recorded them against the day they cleared. On most days those were the same day. On the days they were not, the two books disagreed until the next statement caught up. Nothing was lost. Nothing was double counted. The system was simply using one clock for its own records and inheriting another for everyone else’s.
The fix was not complicated once it was visible: every movement now carries both its own timestamp and the counterparty’s, and reconciliation compares like with like. What made it worth doing was not the few hundred pounds. It was that the same mechanism, applied to a settlement batch rather than a refund, would have produced a five figure difference on a bank holiday weekend, and the team would have spent three days finding it under pressure rather than an afternoon finding it calmly.
Partial settlement, and the temptation to net it off
When a batch half lands, there is always a shortcut available. You can compare totals, spot that the difference equals the sum of the failures, post an adjustment and move on. It closes the books and it takes an hour.
We do not do it, and we ask clients not to either. Netting off destroys the one thing that would have let you find the pattern. Six hundred failed movements is not a number, it is a list, and the list will tell you whether the failures share a payment method, a corridor, an amount band, or a customer. Every time we have insisted on identifying them individually, the list had a shape, and the shape was the actual bug. Twice it was a limit at an intermediary that nobody on either side knew existed.
The number that actually matters
We ask every client the same question early: what is your acceptable discrepancy at the end of a day? The answer we want is zero, expressed in currency rather than percentage. Not because zero is easy, but because any other answer is a decision to tolerate an unknown, and unknowns in a ledger compound.
Zero does not mean nothing ever mismatches. It means nothing mismatches without an explanation attached to it. A timing difference that resolves at settlement is not a discrepancy, it is a known state with an expected resolution date. A difference nobody can account for is a discrepancy of any size, and one penny of it is a reason to stop and look, because the mechanism that produced one penny will eventually produce a larger number.
What this costs
Honestly, more up front. An append only ledger with projections and a real reconciliation harness is perhaps a third more work in the first phase than the version that stores balances and trusts webhooks. We say so in proposals, because the alternative is discovering it in month nine.
What it buys is that the second year is cheap. Systems built this way do not accumulate the sediment of manual corrections and spreadsheet adjustments that eventually makes a finance team distrust the software entirely. We have inherited the other kind, and the first job is always the same: work out which of the numbers can be believed, which usually means rebuilding the history from whatever external records still exist. That is far more expensive than building it properly, and it is done under pressure.
Questions worth asking your team
- If we reprocessed last Tuesday from the raw inputs, would we get the identical ledger?
- Where is the list of transactions that did not match, and who looked at it this morning?
- What happens if our processor sends the same notification twice, four hours apart, with the second one stale?
- Can we produce, for any balance, the complete list of movements that produced it?
- Which of our totals are stored, and which are derived?
If the answers come back quickly and specifically, the system is probably sound. If they come back as “it has not happened yet”, it has, and nobody noticed.
More on how we build in this sector on our financial services page, and the wider argument about correctness under load in what has to be true at any volume.
