Fault Tolerance & Redundancy
Master SLA/SLO/SLI definitions, Active-Active vs Active-Passive redundancy patterns, single point of failure (SPOF) elimination, graceful degradation, and chaos engineering with Netflix Chaos Monkey.
Master SLA/SLO/SLI definitions, Active-Active vs Active-Passive redundancy patterns, single point of failure (SPOF) elimination, graceful degradation, and chaos engineering with Netflix Chaos Monkey. This hands-on tutorial focuses on practical implementation of fault tolerance & redundancy concepts.
Fault Tolerance & Redundancy
Fault tolerance is the ability of a system to continue operating correctly when one or more of its components fail. In distributed systems, failures are not exceptional events — they are a constant, predictable reality. The question is not if something will fail, but when, and whether your system is designed to handle it.
[!IMPORTANT] At Google scale, with ~1 million servers, a server failing every hour is statistically certain. Systems must be designed assuming hardware failure is normal. "Design for failure" is not optional — it is the foundation of reliability.
SLA, SLO, and SLI
Active-Active vs Active-Passive Redundancy
| Active-Active | Active-Passive | |
|---|---|---|
| Resource utilization | ✅ 100% (both serve traffic) | ❌ 50% (standby idle) |
| Failover time | ✅ Instant (LB reroutes) | ⚠️ 30s–2min (detect + promote) |
| Complexity | Higher (conflict resolution) | Lower (single primary) |
| Cost | Same hardware, better ROI | Pay for idle standby |
| Best for | Stateless services, web servers | Databases, stateful services |
Identifying Single Points of Failure (SPOF)
Chaos Engineering
Chaos Engineering is the practice of deliberately introducing failures to find weaknesses before they cause real outages:
[!TIP] Netflix's Chaos Monkey (now open-sourced as the Simian Army) randomly terminates production EC2 instances during business hours. This forces engineering teams to build resilient systems — because flaky services get terminated constantly until they're fixed.
Quiz
Quiz
Question 1 of 2Your service has an SLO of 99.9% uptime per month. In one month (~43,800 minutes), how many minutes of downtime are you allowed before breaching your SLO?
Key Takeaways
✅ SLI is what you measure; SLO is your target; SLA is your customer commitment (SLA < SLO).
✅ Error budget = allowed downtime per period; spending it wisely enables faster innovation.
✅ Active-Active is best for stateless services; Active-Passive for stateful services like databases.
✅ Every single-instance component is a SPOF — systematically identify and eliminate them.
✅ Graceful degradation means failing partially (slower/reduced features) rather than completely.
✅ Chaos Engineering deliberately breaks systems in controlled ways to find weaknesses before customers do.
Official Resources
- 📖 Google SRE Book — Embracing Risk — Error budgets, SLO philosophy
- 📖 Netflix Chaos Engineering — Chaos Monkey and the Simian Army
- 📖 AWS Fault Injection Simulator — Managed chaos engineering on AWS
- 📖 Chaos Engineering Principles — The official principles
What's Next?
You know how to handle component failures — but what about protecting your system from being overwhelmed by too many requests?
Next: Rate Limiting & Throttling — Token bucket, leaky bucket, and how to protect your APIs from abuse and overload.