AGI, Existential Risk, and Asimov's Laws
AGI and Existential Risk
The Concern
Artificial General Intelligence (AGI) refers to algorithms that match or exceed human cognitive ability across a broad range of tasks, rather than being specialised to one domain. The existential risk concern is that a superintelligent system, if not carefully aligned with human values, could cause harm on a catastrophic scale.
Academic Research Status
The MLRD lectures present this as a serious academic research topic, studied by institutions such as:
- CSER: Cambridge Centre for the Study of Existential Risk.
- CFI: Leverhulme Centre for the Future of Intelligence (also at Cambridge).
These are not fringe organisations; they are established academic research centres. The syllabus suggests that whilst AGI may be distant, the safety challenges it poses warrant investigation now.
Current Risks Without AGI
The lectures note that real-world risks from autonomous algorithms exist today, without requiring general intelligence:
- Autonomous stock trading: algorithms making split-second financial decisions without human oversight have caused flash crashes.
- Load balancing: automated power grid management can create cascading failures if not robust to edge cases.
- Autonomous vehicles: imperfect perception systems making life-or-death decisions in real time.
The point: the ethical and safety challenges of AI are not science fiction. They are present in deployed systems now, just at smaller scale.
Asimov’s Laws of Robotics
The lectures present Asimov’s Laws as a philosophical framework, not a technical solution. They illustrate the difficulty of formally encoding ethics.
The Laws
Zeroth Law (added later):
A robot may not harm humanity, or through inaction allow humanity to come to harm.
First Law:
A robot may not injure a human being, or through inaction allow a human being to come to harm.
Second Law:
A robot must obey orders given by humans, unless these conflict with the First Law.
Third Law:
A robot must protect its own existence, unless this conflicts with Laws 1 or 2.
The Hierarchy
The laws form a priority chain: Zeroth > First > Second > Third. A robot must violate a lower law to uphold a higher one.
The Zeroth Law Paradox
The Zeroth Law creates an ethical paradox: it permits harming individual humans “for the greater good” of humanity as a whole. This is utilitarian reasoning encoded as an absolute rule, and it conflicts with the First Law’s absolute prohibition on harming individuals. The tension between these laws is deliberately explored in Asimov’s stories to show that even simple rules produce complex, ambiguous outcomes when applied to real situations.
Why They Are Not a Technical Solution
The laws assume that a machine can:
- Detect what constitutes “harm” (a deeply ambiguous concept).
- Predict the consequences of its actions and inactions (requires perfect world knowledge).
- Resolve conflicts between laws (the Zeroth/First tension).
- Interpret vague terms like “humanity,” “harm,” and “orders” in context.
None of these are currently solvable. The laws are a literary device for exploring ethical dilemmas, not a blueprint for safe AI.
Summary
| Concept | Detail |
|---|---|
| AGI | General intelligence matching or exceeding human ability |
| Existential risk | Superintelligent systems causing catastrophic harm |
| CSER | Cambridge Centre for the Study of Existential Risk |
| CFI | Leverhulme Centre for the Future of Intelligence |
| Asimov’s Laws | Literary framework, not a technical solution |
| Zeroth Law paradox | Permits harming individuals for the “greater good” |
| Key lesson | Formally encoding ethics is extremely difficult |
Past paper questions: y2024p3q9