Testing: The Pyramid, Mocking, Coverage, Mutation, and TDD

Why Test?

Testing is not a clerical exercise or a box-ticking activity. It serves four distinct purposes:

1. Confidence: it provides evidence that the system behaves correctly, and that recent changes have not broken existing functionality (regression).
2. Documentation: test cases are executable specifications of how a function or module is meant to behave. They describe the contract more precisely than any prose comment.
3. Enabling refactoring: you cannot safely restructure code without a suite of tests that proves external behaviour was preserved. Refactoring without tests is just breaking things with confidence.
4. Improving design: code that is difficult to test often reveals poor architecture — tight coupling, hidden dependencies, global mutable state. Writing tests forces modular, composable design.

Manual vs Automated Testing

	Manual Testing	Automated Testing
Pros	Exploratory; catches visual and UX glitches (“this button is misaligned on mobile”); applies human intuition to find edge cases no script would think of	Fast; repeatable; runs on every commit without human intervention; scales to millions of test cases; produces reproducible results
Cons	Slow; unrepeatable (humans miss things differently each time); does not scale; expensive in person-hours	High initial setup cost; tests themselves need maintenance as the system evolves; cannot judge whether a UI is “ugly” or “confusing”

The goal is not to eliminate manual testing — it is to automate everything that can be automated, freeing human testers to do the creative, exploratory work that only humans can do.

The Testing Pyramid

Layer	Scope	Speed	Number	Example
Unit Tests (base)	Test a single function or class in complete isolation. All external dependencies (database, network, file system) are mocked.	< 10 ms per test	Many (thousands)	assert sum(2, 3) == 5
Integration Tests (middle)	Test the boundary between two or more modules, with real implementations of some dependencies (a test database, a local HTTP server running on a random port).	100–1000 ms	Some (hundreds)	Seeds a PostgreSQL container, calls createUser(), then queries the DB directly to verify the row exists
End-to-End (E2E) Tests (top)	Test the complete system from the user’s perspective — a browser navigating pages, a mobile app making real API calls.	Seconds to minutes	Few (tens)	Selenium script: open Chrome, log in as “Alice,” click “Checkout,” verify the confirmation page appears

As you move up the pyramid: cost increases, speed decreases, isolation decreases (a failing E2E test tells you something is wrong but not where — it could be the front end, the API, the database, the network, or any interaction between them), and confidence that the system works as a whole increases.

Mocking and Stubbing

Unit tests must be fast and isolated. If your function calls an external payment API, you cannot call the real API in a unit test — it would be slow, costly (transaction fees), non-deterministic (network failures), and potentially dangerous (real money moving).

Instead, you replace the real dependency with a test double:

Type	Behaviour	Used to verify…
Stub	Returns a hardcoded, canned response regardless of input. paymentGateway.charge() always returns {"status": "success"}.	That the code under test handles the response correctly — it lets the test run without a real dependency.
Mock	Records every call — how many times, with what arguments. After the test, you assert that the mock was called exactly once with the expected arguments.	That the code under test used the dependency correctly — not just that it handled the response, but that it issued the right request at the right time.

Exam distinction: a Stub answers the question “did the code work given this response?” A Mock answers the question “did the code even call the right thing the right number of times?” Getting these confused is a classic exam trap.

Code Coverage ≠ Correctness

Code coverage measures the percentage of source lines (or branches, or paths) that are executed by the test suite. It is a useful metric for identifying dead code and completely untested regions.

It is not a measure of correctness.

The classic trap:

def average(scores):
    return sum(scores) / len(scores)

def test_average():
    assert average([10, 20]) == 15

This single test achieves 100% line coverage — every line of average is executed. But average([]) crashes with a ZeroDivisionError. Coverage tracked which lines ran; it did not track which edge cases were verified.

High coverage is a necessary condition for a good test suite, but it is never sufficient. You can have 100% coverage and still have bugs; you cannot have 0% coverage and have a reliable system. The metric is a floor, not a ceiling.

Mutation Testing: Testing Your Tests

If high coverage only proves code was executed, not verified, how do you measure test quality? Mutation testing automatically injects small, artificial bugs (“mutants”) into the source code and checks whether the existing test suite catches them.

Examples of mutations:

• Change > to >= (off-by-one sensitivity).
• Change + to - (arithmetic sensitivity).
• Delete a method call entirely (side-effect sensitivity).
• Swap true and false in a conditional.

After injection, the test suite is re-run:

• If the tests fail, the mutant is killed — the tests were strong enough to detect the injected bug. Good.
• If the tests still pass, the mutant survived — the tests executed the mutated line but did not have an assertion that would detect the change. Bad.

A high survival rate indicates that the test suite is executing code without actually verifying its behaviour — coverage without meaningful assertion. Mutation testing is computationally expensive because the entire test suite must be re-run against every mutant, so it is typically run periodically rather than on every commit for large codebases.

Test-Driven Development (TDD): Red-Green-Refactor

TDD inverts the traditional sequence. Instead of writing code then testing it:

1. Red: write a failing test for the new behaviour. Run it; it fails because the code does not exist yet. This proves the test is actually testing something (a test that never fails is a test that never catches bugs).
2. Green: write the minimum code to make the test pass. No more. It may be ugly, inefficient, or incomplete — the goal is to satisfy the test’s contract as quickly as possible.
3. Refactor: clean up the code — extract methods, rename variables, improve the algorithm — while keeping the tests green. The suite ensures that external behaviour is preserved during restructuring.

Result: 100% test coverage by construction (every line of production code was written to satisfy a failing test), and the code is inherently designed for testability because the tests existed first.

Expected Learning

• Describe the Testing Pyramid and explain why the layers are arranged as they are (cost, speed, isolation, confidence).
• Distinguish a Stub from a Mock, and explain when each is appropriate.
• Explain why 100% code coverage does not guarantee correctness, with a concrete example.
• Define mutation testing, and distinguish a killed mutant from a survived mutant.
• Describe the Red-Green-Refactor cycle and explain what each phase achieves.
• Explain why testing enables safe refactoring.

---

Past Paper Questions

Example Sheet 2, Question 2 directly tests mutation testing: why high coverage can coexist with bugs, what killed vs survived mutants mean, and why mutation testing is too expensive for every CI build. Example Sheet 2, Question 3 tests mocking/stubbing with a Stripe API scenario and explores how discrete state both helps and harms testing.