Example Sheet 2: Quality, Evolution & AI — Worked Answers | Software and Security Engineering

Q1. VCS and Conflict Resolution

(a) Alice renamed get_name() to get_full_name(); Bob added a new method get_initials() in a separate part of the file. Is there a merge conflict?

No textual merge conflict — Git merges line-by-line, and their changes touch different lines. However, there may be a semantic conflict: all other callers of get_name() throughout the codebase will no longer compile or will call a now-non-existent function. Git cannot detect this; only the test suite can.

(b) If Bob also modified code inside get_name() at the same lines Alice renamed, how does a distributed VCS (Git) handle this?

Git detects that both branches touched the same lines and flags a merge conflict. It marks the conflicted region in the file and pauses for a human to resolve it. Git provides both versions (Alice’s rename, Bob’s internal change) and the common ancestor. A centralised VCS (SVN) handles this similarly in principle, but typically requires a network round-trip and offers less local tooling to reason about the conflict.

(c) Branching strategy for 30 developers under tight deadlines:

Trunk-Based Development with short-lived feature branches (< 1 day), merged frequently via small Pull Requests, backed by a strong CI pipeline that runs the full test suite on every merge. This avoids the “Merge Hell” of long-lived Git-Flow-style branches accumulating divergence, and surfaces conflicts while they are still small.

Q2. QA: Mutation Testing

(a) The team has 95% code coverage but bugs still reach production. Why, and how does mutation testing expose the weakness?

High coverage proves lines were executed, not that tests verify correct behaviour (cf. the average([]) coverage trap). Mutation testing injects artificial bugs (mutants) and checks whether tests catch them. If many mutants survive, it proves the assertions are too weak for the code paths being “covered.”

(b) Killed vs survived mutants:

Killed: the test suite failed (caught the injected bug) — the tests were strong enough. Good.
Survived: all tests still pass despite the injected bug — that logic path has no effective assertion. Bad.

(c) Why not run mutation testing on every CI build?

For every mutant, the entire relevant test suite must be re-run. A large codebase generates thousands of mutants per file, multiplying test-suite runtime by orders of magnitude. This is computationally too expensive to run on every commit; it is typically run periodically or on a sampled subset.

Q3. Testing in the Real World: Isolation

(a) Why not use the real Stripe API in unit tests?

Real API calls are slow (network round-trip), costly (transaction fees, rate limits), non-deterministic (network failures, sandbox maintenance), and could have real side effects (unintended charges). This violates the core unit-test requirement of being fast, isolated, and repeatable.

(b) Stub vs Mock:

Stub: returns a hardcoded value — charge() always returns {"status": "success"}. Used to let the test run without a real dependency.
Mock: records interactions — how many times was charge called, with what arguments? Used to verify the code under test used the dependency correctly, not just that it handled the response.

(c) Discrete state: helpful or harmful for testing a payment system?

Helpful: “Success” and “Insufficient Funds” are exact, enumerable states you can stub deterministically and assert against precisely — unlike testing a continuous physical system (e.g. a refrigerator’s temperature controller). Harmful: a single incorrect branch can silently flip a transaction from “Insufficient Funds” to “Success” (or vice versa) with no gradual warning. Exhaustively testing every discrete combination of account states, payment amounts, currency conversions, and retry scenarios is not feasible, so edge-case combinations are easily missed.

Q4. Reliable Delivery: Deployment Strategies

(a) Blue-Green vs Canary for a high-risk database migration:

Blue-Green: runs two full environments, switches all traffic at once. Instant rollback. But 100% of users are immediately exposed to any migration defect.
Canary: exposes new version to a small traffic percentage first and expands gradually while monitoring. For a high-risk database migration, Canary is more appropriate because it limits the blast radius to a small fraction of users, while monitoring reveals data-corruption or performance issues before they affect everyone.

(b) Feature Flags and Progressive Delivery:

Feature Flags decouple deployment (code reaching production) from release (feature visible to users). This enables Progressive Delivery: the same deployed build can be turned on for developers, then beta users, then everyone, and instantly disabled (“emergency shutoff”) without a code rollback.

(c) Error Budget throttling releases:

The Error Budget is $100\% - \text{SLO}$ . If the budget is exhausted (too many incidents this month), all risky new feature deployments freeze and the team redirects effort to stability work until the error rate recovers. Deployment velocity is directly throttled by reliability performance — a self-regulating mechanism.

Q5. The Maintenance Reality

(a) Why freezing a successful 20-year-old tax system is impossible:

Lehman’s Law of Continuing Change: the system must be continually adapted or it becomes progressively less satisfactory. Tax regulations, security standards, and operating environments change. Even if the code is untouched, it decays through Bit Rot — dependent libraries are deprecated, TLS versions are retired, OS APIs change. Freezing it today guarantees it will be non-functional within a few years.

(b) Classify the maintenance tasks:

(i) Fixing a rounding error in tax calculation → Corrective (fixing a discovered bug).
(ii) Adding integration with a new government tax API → Adaptive (responding to a changed external environment — new regulations).
(iii) Rewriting a data-fetching loop to be more efficient → Perfective (improving performance without changing behaviour).
(iv) Migrating the on-premises mainframe to the cloud → Adaptive (the primary driver is responding to a changed deployment environment, though there are perfective elements in the cloud-native re-architecture).

(c) Why investing in a test suite is economically rational for a 20-year system:

The 80/20 rule: ~80% of lifetime cost is maintenance. A comprehensive test suite saves money on every one of the thousands of future corrective, adaptive, and perfective changes by making them safe and fast — enabling confident refactoring and catching regressions instantly. The upfront investment is small relative to the compounding savings over two decades.

Q6. SRE and Chaos Engineering

(a) Chaos Monkey’s philosophy:

Deliberately and randomly kill production servers during business hours (when engineers are present and watching). This forces systems to be designed with redundancy and automatic recovery before a real failure occurs. It shifts the operational mindset from maximising MTBF (preventing all failure) to minimising MTTR (surviving failure gracefully). “If your server might die at 2 p.m. on a Tuesday, make sure it doesn’t matter.”

(b) Why is saturation harder to measure than latency?

Latency is a direct, unambiguous per-request timing — you measure it. Saturation requires knowing the resource’s true maximum capacity (which varies with workload, caching state, and hardware configuration) to express “how full” the system is as a meaningful percentage. There is no universal “100% saturated” reading; the ceiling shifts.

(c) Monitoring vs Observability:

Monitoring addresses known unknowns — pre-defined dashboards telling you that a known metric has crossed a threshold (e.g. “CPU is at 99%”).
Observability addresses unknown unknowns — the ability to ask arbitrary new questions of logs, metrics, and traces after an incident, to understand why something unexpected happened, without having predicted that specific failure mode in advance.

Q7. Legacy Evolution: The Strangler Fig

(a) Why a Big Bang rewrite of a successful system is high-risk:

It halts all new feature delivery for an extended, uncertain period — cf. Netscape’s 3-year rewrite that let Internet Explorer capture 90% market share. It risks silently dropping “undocumented” behaviour that real users depend on (Hyrum’s Law), since the new system is built from a fresh, incomplete understanding of the old one. The “all-or-nothing” cutover concentrates all migration risk into a single high-stakes event.

(b) Describe the Strangler Fig approach for a COBOL mainframe:

Place a proxy/facade in front of the legacy COBOL system.
Build new functionality as a separate, modern microservice alongside the legacy.
Route specific calls/traffic to the new service; the rest continues to hit the mainframe.
Incrementally repeat, migrating more and more functionality until the legacy receives no traffic and can be safely decommissioned.

(c) How Hyrum’s Law complicates the migration:

Over 30 years, downstream consumers have come to depend on undocumented quirks — specific rounding behaviour of the COBOL system, edge-case output ordering, side effects of a “temporary” fix from 1997. The new system cannot be built purely from the documented specification. It must be validated against real historical behaviour via characterisation (Golden Master) tests — recording the current system’s output and asserting the new system produces identical results — rather than a clean-room reimplementation of the intended spec.

Q8. Software Archaeology and API Design

(a) Three steps of Software Archaeology:

git blame — see who changed each line and when. Trace the change history to understand a function’s evolution.
git log -S "functionName" — search the entire commit history for when a specific string was added or removed, recovering intent from the commit message.
Trace linked issue-tracker IDs in commit messages (e.g. FIX-123) back to the original bug report or feature request, preserving the business context that motivated the change. Then write characterisation tests to lock in current behaviour before touching anything.

(b) SemVer: process(data) → process(data, options) — what version bump?

Adding a required parameter is a breaking structural change. Existing callers using the single-argument signature will not compile. The new version requires a MAJOR bump, e.g. 2.4.1 → 3.0.0. Unless options is given a default value making the two-argument signature backward-compatible — in which case it is a MINOR bump to 2.5.0. You must state your assumption: “as written, without a default, this is MAJOR.”

(c) Why is an API a contract?

An API promises: given input $X$ , I return output $Y$ . Every consumer builds on that promise. When you break the contract (a breaking change without warning), every downstream consumer’s code may stop compiling or functioning correctly, breaking their CI/CD pipelines and production systems. One change propagates across the entire ecosystem of dependents (cf. Hyrum’s Law, Left-Pad).

Q9. API Evolution & Deprecation

(a) Why turning off the XML endpoint tomorrow is a bad idea:

It is an uncommunicated breaking change. Existing customer integrations built against XML would fail instantly with no warning, breaking their production systems and CI/CD pipelines. Per Hyrum’s Law, however officially “deprecated” XML may be, real customers depend on it continuing to work until they choose to migrate.

(b) Describe the 3-step deprecation path:

Announce: publish a notice that XML support will be removed, with a timeline. Run both XML and JSON in parallel (MINOR version bump — backwards-compatible).
Deprecate: mark the XML endpoint with a Deprecation HTTP header and documentation warnings. Give customers 6–12 months to migrate.
Remove: in a new MAJOR version, delete the XML endpoint after the migration window has passed and usage has dropped to near-zero.

(c) How an abrupt XML removal cascades through consumers’ CI/CD pipelines:

Any customer whose integration tests call the old endpoint will experience failing builds the moment your change ships. Their Quality Gate fails; their build is rejected. You have propagated your internal API change into an involuntary production incident for every customer — simultaneously.

Q10. AI-Augmented Engineering

(a) Why did the AI suggest an outdated, vulnerable library? Who is responsible?

The AI’s training data reflects historically common code (old tutorials, Stack Overflow snippets using an older package version). It pattern-matches to statistically frequent solutions, not the most current or secure one — a training-data bias / hallucination risk. Responsibility lies with the developer who committed the code: “you own every line you commit.” Code review, dependency scanning (Dependabot/Snyk), and understanding what was shipped are the developer’s responsibility, not the AI’s.

(b) Compare a 2010 junior engineer with a 2026 junior engineer:

A 2010 junior spent significant time on manual boilerplate, syntax lookup, and low-level implementation, gradually building system-level understanding through repetition. A 2026 junior orchestrates AI to generate that boilerplate rapidly. The cognitive skillset shifts from writing syntax to critically reviewing, verifying, and directing AI output — requiring earlier, faster development of architectural judgement and domain knowledge, precisely the skills the “training ladder” used to build gradually over years of manual practice.

(c) What is the Reasoning Ceiling?

LLMs excel at tasks with a clear feedback loop (does it compile? do tests pass?) and pattern-matching against vast training data. They lack: long-horizon strategic judgement across ambiguous, conflicting stakeholder priorities; accountable ownership of consequences; and tacit, cross-domain organisational context a Senior Architect accumulates over years. AI can accelerate execution but cannot currently replace the judgement, accountability, and cross-domain synthesis of a Senior Architect.

Q11. The Human in the AI Loop

(a) What is a context window, and how did it cause the database migration scripts to be missed?

A context window is the finite amount of text (code, files, conversation) an LLM can consider at once. The AI correctly updated all files within its context window (the main codebase) but had no visibility of the migration scripts in a separate, unreferenced directory — it did not know they existed or needed updating. Context window limits are a real architectural constraint.

(b) AI Review Checklist for a senior engineer before merging AI-generated code:

Does it compile and pass the full test suite, including integration tests spanning the files touched?
Were all related artefacts updated — migrations, configuration, documentation, infrastructure-as-code? Explicitly search for cross-cutting concerns the diff might have missed.
Does the reviewer understand every line well enough to maintain it (“you own every line you commit”)?
Are there hallucinated dependencies/APIs, or newly introduced libraries with known CVEs?
Does the change match existing architectural conventions, or does it introduce a “clever hack” that violates long-term design goals?

(c) Why Software Archaeology remains essential in the AI era:

AI can summarise what a file currently does — but summarising current behaviour is not the same as understanding why the code is the way it is. The historical intent, prior incidents, and business context behind a design decision (captured in git blame, linked issue trackers, and tribal knowledge) are not recoverable from the file’s text alone. This context determines whether a proposed AI change is safe, and it is exactly why Software Archaeology remains a human engineering discipline.