AI as a Coder: Capabilities and Risks | Software and Security Engineering

Note: this is a high-velocity area; breakthroughs occur frequently. Exam answers should focus on enduring engineering principles (verification, ownership, architecture) rather than which specific tool is currently fashionable.

How AI Assists Coding

LLM-based coding tools have capabilities that go far beyond traditional IDE autocompletion:

Semantic autocompletion vs syntactic IntelliSense. Traditional LSP-based completion knows that user. should be followed by a member of the User class. LLMs understand intent: a comment like // Calculate the Haversine distance between two GPS coordinates triggers the model to generate the full mathematical formula and implementation — it understands the semantics of the task, not just the syntax.
Project-level context (RAG for code). Modern tools read your package.json, your existing utility functions, your naming conventions, and your code style. They generate code that matches your codebase’s idioms, not a generic Stack Overflow solution.
Boilerplate elimination. Estimates suggest 40–60% of professional code is boilerplate: ORM schemas, DTOs, API route definitions, test scaffolding, CRUD endpoints. AI can generate this from a description, reducing “time to Hello World” from hours to seconds.
Rapid prototyping. An AI can generate an entire directory structure, Dockerfile, database schema, and API endpoints in a single pass. This enables rapid hypothesis testing: build a throwaway prototype in an afternoon to validate an idea before investing engineering resources.
Language translation. Switching languages involves “syntactic tax” — re-learning idioms, standard-library functions, and ecosystem conventions. AI handles much of this tax, making it practical to rewrite a Python proof-of-concept in Rust for performance, or migrate a C codebase to a memory-safe language.
Test generation. AI can generate comprehensive test suites — happy paths, edge cases, and mocks — from a function signature and a description. However, it tests what the code does, not necessarily what it should do. The test may faithfully verify incorrect behaviour.
AI fuzzing. Unlike random-byte fuzzing, AI-generated fuzz inputs are plausible but malformed — e.g. a JSON payload with a circular reference, or an HTTP request with valid syntax but logically contradictory headers. This targets deeper validation bugs.
Self-healing CI/CD. An AI agent reads a failing test’s error log, identifies the failing line, and proposes a fix commit. This acts as a “night watchman”: when the build breaks at 3 a.m., a candidate fix is ready by the time engineers wake up.
Debugging and anomaly detection. AI can correlate disparate information — “Service A expects ISO-8601 timestamps; Service B sends Unix timestamps” — across terabytes of logs, identifying cross-service mismatches that a human debugging session might not spot for hours.
Refactoring and semantic search. AI can restructure high-cyclomatic-complexity functions and perform semantic search across a 10-million-line codebase: “show me every place we calculate VAT.”

Risks of AI-Assisted Coding

These are not hypothetical — they are observed failures of current-generation tools. You must be able to name the specific failure mode, not just say “AI makes mistakes.”

Hallucinations

LLMs can confidently generate code that calls libraries that do not exist, API methods that were never implemented, or parameters in the wrong order. The model is pattern-matching against its training data, which includes documentation for old versions, deprecated APIs, and — occasionally — entirely fictional packages invented in someone’s blog post. The code looks correct and plausible, but does not compile.

Technical Debt of Ignorance

If you submit AI-generated code that you do not understand, you cannot maintain it. AI often produces “clever” solutions — a regex that handles every edge case but is incomprehensible, a concurrency pattern that passes tests but has a subtle race condition under specific timing — that violate long-term architecture and create maintenance burdens. You own every line you commit, regardless of its origin.

Not All Models Are Equal

A small, fast, locally-running model may produce quick-but-incorrect code. A larger, slower, cloud-based model may have better reasoning but higher latency and cost. The trap: assuming that because the model produced 10 correct answers, the 11th will also be correct. LLMs do not “know” when they are wrong — they produce the most statistically probable output, which is not the same as the correct output.

Algorithmic and Training-Data Bias

LLMs are trained on public code (GitHub, Stack Overflow). This training data reflects biases:

Popularity bias: the model favours well-known libraries and patterns, even when they are sub-optimal for the specific use case.
Security bias: the training data includes historically common-but-insecure patterns (e.g. SQL string concatenation instead of parameterised queries; outdated cryptographic primitives).
Demographic bias: in socio-technical systems (e.g. a loan-approval algorithm), the model may reflect biases in the training data that produce discriminatory outcomes.

Mitigation: Generator/Verifier (Critic) Architecture

One effective mitigation strategy is adversarial:

A Generator model writes the code.
A separate Verifier (Critic) model — which may use a different architecture or different training data — tries to find bugs, security flaws, style violations, or hallucinated APIs in the generated output.

The verifier acts as an automated code reviewer, substantially reducing (though not eliminating) the rate of hallucinations reaching production. This is conceptually analogous to pair programming: two independent “perspectives” on the same code catch more defects than one.

Expected Learning

Name at least five distinct capabilities AI brings to the coding workflow, with a concrete example of each.
Distinguish hallucinations, technical debt of ignorance, model quality variance, and training-data bias as distinct AI coding risks.
State the rule that the developer owns every line they commit, regardless of origin.
Describe the Generator/Verifier architecture and explain how it mitigates hallucinations.

Past Paper Questions

Example Sheet 2, Question 10(a) presents a scenario where AI generates code using an outdated, vulnerable library version — testing your ability to name the specific failure mode (training-data bias / hallucination) and to assign responsibility correctly (the developer, not the AI). Example Sheet 2, Question 11(b) asks you to design an AI review checklist for a senior engineer.