SRE, Chaos Engineering, and Supply-Chain Security
Site Reliability Engineering (SRE)
Ben Treynor Sloss (Google) defined SRE as:
SRE is what happens when you ask a software engineer to design an operations team.
The core philosophy: treat operations as a software problem. If a task is repetitive, automate it. If a failure mode is predictable, build a system that survives it rather than hiring more people to wake up at 3 a.m. and manually restart servers.
The 50% Rule
Google SREs spend a maximum of 50% of their time on toil — manual, repetitive, automatable operational work (handling alerts, restarting services, running deployment scripts). The other 50%+ is spent on engineering projects that reduce future toil — building self-healing systems, improving monitoring, automating runbooks.
If toil exceeds 50%, the SRE team escalates: management must either reduce the operational burden (by investing in automation) or accept that reliability will degrade.
MTBF vs MTTR
Traditional operations focuses on preventing failure: maximise MTBF (Mean Time Between Failures) through careful change management, redundancy, and hardened infrastructure.
Modern SRE accepts that failure is inevitable — especially in large, distributed systems where hardware, networks, and third-party services can (and do) fail unpredictably. The focus shifts to MTTR (Mean Time To Recovery): how quickly can the system detect a failure and self-recover, ideally without human intervention?
A system with a low MTBF but an MTTR of 30 seconds (automatic failover) is more reliable in practice than a system with a high MTBF but an MTTR of 4 hours (waiting for an on-call engineer to drive to the data centre).
Blameless Post-Mortems
When an incident occurs in production, the instinctive reaction is to identify who caused it. The SRE approach is the opposite: a blameless post-mortem asks why the system allowed the human to make this mistake, rather than who do we blame.
The principle: in a well-engineered system, a single human error should not be able to cause a catastrophic failure. If an engineer can take down production by running a command, the system lacked sufficient safeguards — the failure is systemic, not personal. A blameless post-mortem produces:
- A timeline of the incident: what happened, in what order, observed by whom.
- Root causes framed as system weaknesses: “the deployment script did not require confirmation before running against the production database.”
- Action items that are concrete, assigned, and tracked: “add a confirmation prompt that requires explicit typing of the target environment name.”
The alternative — a blame culture — causes engineers to hide mistakes, delay incident reports, and avoid risky but necessary changes. A blameless culture causes engineers to report incidents immediately, share knowledge openly, and build safeguards that make the next incident less likely and less severe.
Monitoring vs Observability (Revisited)
| Monitoring | Observability | |
|---|---|---|
| Addresses | Known unknowns — you set up an alert for “CPU > 90%” because you predicted CPU might be a bottleneck | Unknown unknowns — you can ask arbitrary new questions of the system after an incident, for failure modes you did not predict |
| Pre-requisite | You must know in advance what to monitor | You must have rich telemetry (metrics, logs, traces) that you can explore post-hoc |
| Answers | ”Is something wrong?" | "What specifically went wrong, and why?” |
The Four Golden Signals
Google’s SRE book identifies four metrics that cover the vast majority of service-health questions:
| Signal | Definition | Why it matters |
|---|---|---|
| Latency | Time taken to service a request | User experience. High latency → slow pages → user abandonment. |
| Traffic | Demand on the system (e.g. requests per second) | Capacity planning. A spike in traffic may saturate resources. |
| Errors | Rate of failed requests (e.g. HTTP 5xx, timeouts, or application-level failures like “payment declined” when it should have succeeded) | Direct measure of correctness. Errors that are invisible to the HTTP layer (e.g. a 200 OK response with a “null” body) still need tracking. |
| Saturation | How “full” the system is — CPU utilisation, memory pressure, disk I/O queue depth | Leading indicator of imminent failure. Saturation often rises before latency and errors spike. |
Saturation is harder to measure than latency because latency is a direct, unambiguous per-request timing — you just measure it. Saturation requires knowing a resource’s true maximum capacity (which varies with workload, caching state, and hardware configuration) to express “how full” the system is as a meaningful percentage.
Chaos Engineering
The discipline of experimenting on a software system to build confidence in its capability to withstand turbulent conditions in production.
Chaos Monkey (Netflix, 2010): as Netflix migrated from stable on-premises data centres to the “unreliable” cloud (AWS), they knew EC2 instances would disappear randomly. Instead of hoping they wouldn’t, they built a tool that deliberately killed production servers during business hours. If your server might die at 2 p.m. on a Tuesday, engineers build services that survive it. Chaos Monkey forced this by design.
The tool expanded into the “Simian Army”: Chaos Gorilla kills an entire AWS availability zone. Chaos Kong kills an entire AWS region. Each level tests whether the system’s redundancy is real or merely documented.
The Chaos Engineering method:
- Define the Steady State — a measurable indicator of normal behaviour (e.g. error rate < 0.1%, 95th-percentile latency < 200 ms).
- Form a Hypothesis — “if we terminate the primary database, the read replica will be promoted and the cache will absorb reads during the failover.”
- Introduce a controlled Experiment — actually terminate the primary database (during business hours, with engineers watching).
- Verify whether the steady state was maintained. If it was, confidence increases. If it was not, you found a real vulnerability before a real failure exploited it.
Software Supply-Chain Security
Modern applications depend on hundreds of third-party libraries — often pulled automatically by package managers (npm, PyPI, Maven, Cargo). A transitive dependency is your dependency’s own dependency — a library you never explicitly chose, but is pulled in automatically.
Left-Pad (2016)
A developer deleted an 11-line npm package called left-pad (which padded strings with spaces on the left — truly trivial functionality). Within hours, thousands of projects failed to build — including React, Babel, and countless others. Many developers did not even know they had a transitive dependency on left-pad.
Lesson: you are responsible for the security and availability of every library in your dependency tree, not just the ones you explicitly chose. An 11-line package deleted in a personal dispute can take down your entire CI/CD pipeline.
Log4Shell (2021)
A critical vulnerability in log4j, a ubiquitous Java logging library, allowed attackers to execute arbitrary code on a server simply by sending a crafted string to be logged — via a chat box, a search bar, or an HTTP header. Companies spent weeks manually searching their systems to determine whether they even used log4j anywhere in their dependency trees.
The solution: an SBOM (Software Bill of Materials) — a machine-readable inventory of every component and library in a software system. When a new CVE (Common Vulnerabilities and Exposures) is published, you can query your SBOM to determine instantly whether you are affected, rather than performing a manual audit.
Tools like Dependabot and Snyk automate this: they scan your dependency tree, cross-reference it against known CVEs, and open automated pull requests to upgrade vulnerable libraries.
Expected Learning
- Define SRE and explain the 50% rule (toil vs engineering work).
- Contrast MTBF-focused (traditional ops) and MTTR-focused (SRE) approaches to reliability.
- Name the Four Golden Signals and explain why saturation is harder to measure than latency.
- Describe Chaos Monkey’s philosophy and the four-step chaos engineering method.
- Explain the Left-Pad incident and what it teaches about transitive dependencies.
- Explain the Log4Shell incident and the role of an SBOM in supply-chain security.
Past Paper Questions
Example Sheet 2, Question 6(a-c) directly addresses Chaos Monkey, the Four Golden Signals, and the monitoring vs observability distinction. These concepts are examinable and may appear in a scenario-based Tripos question about operating a rapidly growing distributed service.