What reliability actually costs at scale
Reliability sounds like a goal everyone agrees on until you ask what it costs. “More reliable” is never free, the bill is rarely where you expect, and the price of each improvement isn’t linear — it climbs steeply the closer you get to perfect. Pretending otherwise is how teams end up over-paying for reliability in one place and under-investing where it actually matters.
On a high-traffic platform in a regulated industry, this trade-off is a constant companion, so it’s worth being honest about it.
The bill is non-linear
Going from “usually works” to “reliably works” is cheap and obviously worth it. Going from very reliable to almost-never-fails is where the cost curve bends. Each additional nine of uptime — 99.9% to 99.99% and beyond — tends to cost disproportionately more than the last, because you’re no longer fixing obvious gaps. You’re engineering against rarer and rarer failure modes, each requiring more machinery to catch.
That doesn’t mean stop. It means know where you are on the curve, because the same effort that buys a huge gain early buys a marginal one later.
The costs that don’t show up on an invoice
The obvious cost is infrastructure — redundancy, failover, spare capacity. But the expensive costs are the quiet ones:
- Complexity. Every reliability mechanism — retries, replicas, fallbacks — is more system to understand, operate and debug. Past a point, the machinery you added for reliability becomes its own source of failure.
- Speed. Effort spent hardening the rare case is effort not spent shipping. A team chasing an extra nine everywhere is a team moving slowly everywhere.
- People. On-call, incident response and the vigilance reliability demands are paid in human attention, and that’s the most finite budget you have.
None of these appear in a cloud bill, which is exactly why they get underweighted.
Choosing the target on purpose
The trap is treating “more reliable” as universally good and applying it evenly. Not every part of a system carries the same consequence. A payment path or a regulatory obligation may genuinely warrant extreme reliability. An internal dashboard almost certainly doesn’t — and spending the same effort on both is just waste wearing the costume of diligence.
So the real work is deciding, deliberately, how reliable each part needs to be, and tying that target to actual user and regulatory impact rather than to a number that feels reassuring. Five nines where it matters; “good enough, and we’ll know if it breaks” where it doesn’t.
The discipline of “good enough here”
The mature move — and the hard one — is saying good enough about parts of your own system. Engineers don’t love it; it feels like settling. But reliability is a budget, not a virtue you maximise everywhere. Spent evenly, it runs out before it reaches the places that genuinely can’t fail.
The teams I trust most aren’t the ones chasing perfection across the board. They’re the ones who know exactly which failures they refuse to allow, invest heavily there, and are honest about consciously accepting the rest. That’s not lowering the bar. That’s knowing where the bar actually belongs.