Designing for the failure case, not the happy path
Most software is demoed on its happy path — the run where every service responds, every input is well-formed and nothing times out. That path is the easy 80% of the work. The other 20%, the part that decides whether a platform is actually dependable, is everything that happens when one of those assumptions breaks.
Working on a high-scale, regulated platform teaches you this quickly: the interesting question is never “does it work when everything is fine.” It’s “what happens when something isn’t.”
The happy path is the part you can take for granted
When a system is small and lightly used, failures are rare enough to treat as exceptions — you fix them when they happen. At scale, that math inverts. A one-in-a-million event happens many times a day. A dependency that’s up 99.9% of the time is down for you, somewhere, constantly. Failure stops being an edge case and becomes a steady-state condition you’re always operating inside.
Once you internalise that, you stop designing for the run where everything works and start designing for the run where something doesn’t.
Failure is a design input, not an afterthought
The shift is from “how do I make this work” to “how does this behave when its dependencies don’t.” That’s a different question, and asking it early changes the design rather than bolting safety on at the end.
In practice it looks like a handful of habits:
- Assume every call can fail or hang. Timeouts, retries with backoff, and circuit breakers aren’t decoration — they’re the difference between one slow dependency and a cascading outage.
- Degrade gracefully. Decide in advance what the system does without each piece. A feature that quietly falls back beats a page that won’t load.
- Make operations idempotent. At scale, things get retried. If a repeat doesn’t double-charge or double-send, whole classes of bug simply can’t happen.
- Contain the blast radius. When something breaks, how much breaks with it? Good boundaries mean a failure stays a small failure.
- Make it observable. You can’t respond to what you can’t see. The failure case needs logs, metrics and alerts as deliberately as the feature does.
None of this is glamorous, and none of it shows up in a demo. It shows up at 3am, or under a load spike, or when a third party has a bad day — which, given enough traffic, is every day for someone.
Reliability is a product feature
It’s tempting to treat all this as engineering hygiene, separate from the product. It isn’t. To the person using the platform, “it works when I need it” is the product. The cleverest feature is worth nothing on the day it won’t load.
So I’d rather a team spend the extra time designing for the failure case than ship a wider happy path that falls over the first time reality doesn’t cooperate. The happy path gets you the demo. The failure path gets you trusted — and at scale, in a regulated setting, trust is the whole game.