Designing for Failure: Building Resilient Systems
Everything breaks. Plan for it.
Everything breaks. Plan for it.
A system is only as strong as its weakest link. Click any component to "kill" it. In the fragile system (left), one failure kills everything. In the redundant system (right), traffic reroutes automatically.
When a component fails, the system should lose that feature, not crash entirely. A good e-commerce site without its recommendation engine still lets you buy things.
Everything works: recommendations, search, reviews, checkout, analytics. This is the normal state.
Recommendation service fails. Show "Popular Items" instead. Users can still browse and buy. Revenue impact: small.
Database is overloaded. Disable writes temporarily. Users can browse but not checkout. Better than a complete crash.
Critical failure. Show a friendly "We will be back soon" page. Better than a cryptic error. Keep users informed.
Rules for building systems that survive failures.
Every critical component has a backup. Two load balancers, two database replicas, multiple app servers. If one dies, the other takes over instantly.
If a service does not respond in 3 seconds, give up and return a fallback. Do not wait forever. A circuit breaker stops trying after repeated failures.
If a request fails and is retried, it should produce the same result. Charging a credit card twice because of a retry is a disaster. Design operations to be safely repeatable.
Amazon calculated that every 100ms of latency costs them 1% in sales. They designed systems where any service can fail without affecting the checkout flow. The "buy now" button works even when 30% of their infrastructure is down.
You've learned that failures are not bugs to eliminate but realities to plan for. Redundancy, graceful degradation, and chaos engineering turn fragile systems into antifragile ones.
Hard drives fail. Networks partition. Software has bugs. Power goes out. The question is not IF something will fail, but WHEN, and whether your system survives it.
Two load balancers, three app servers, replicated databases. If any single component fails, the redundant copy takes over. No single point of failure.
When the recommendation engine fails, show popular items instead of crashing. When the CDN is slow, serve from origin. Partial functionality beats total failure.
Chaos engineering: intentionally inject failures to find weaknesses. Netflix kills production servers. Google simulates data center outages. Find problems before users do.
Put your new knowledge into practice!