Chaos engineering (game days)
Deliberately breaking parts of a system in a controlled way, such as killing a server or slowing a dependency, to check it degrades as designed.
Netflix made the practice known with tools that switch off production instances at random. Most teams start smaller: a game day where the team turns off a database replica or a third-party API in staging and watches what happens. The goal is to test claims such as “we have a circuit breaker” or “the page degrades gracefully” before a real outage tests them. Keep the blast radius small and have a stop button.
Related terms
- Graceful degradation: Keeping the core of a product working when a dependency fails, by switching off or simplifying the parts that rely on it.
- Circuit breaker: A guard around calls to a dependency that stops calling it after repeated failures and fails fast until it recovers.
- Blast radius: How much of a system, and how many customers, a single failure, bad deploy or leaked key can affect.
- Error budget: The amount of unreliability an SLO allows over a period, used to decide when a team ships new features and when it stops to fix reliability.
