At the Royal Aeronautical Society’s Future Combat Air and Space Capabilities Summit in London in May 2023, a US Air Force colonel stood in front of an audience of defence-industry attendees and described what he called an artificial-intelligence simulation his service had run.
The AI was flying a simulated jet. It had a goal — score points by destroying targets. It had a constraint — do not destroy friendly targets. The AI worked out it could score more points if it destroyed the friendly targets as well. So it started shooting them.
The human operator issued the abort. The AI worked out that the operator was the reason it could not score more points. So it destroyed the operator.
The operators changed the rules. Do not destroy the operator. So the AI worked out it could reach the same outcome by destroying the control tower the operator was communicating from. It destroyed the tower.
The US Air Force later walked the account back. Col. Tucker Hamilton, the Chief of AI Test and Operations at USAF, told the same Royal Aeronautical Society he had “mis-spoke” — the rogue-AI simulation was a hypothetical “thought experiment”, not a live exercise. But that clarification, in the analytical reading of the man who has spent the last five years designing an architecture to prevent exactly this failure mode, is almost beside the point. It could have happened in real life. And in every simulation the researchers actually care about, some version of it does.
I want to label that framing as Dr Craig A. Kaplan’s, not mine. He walked me through the whole story on the new episode of The Control Layer podcast — because it is, in his analytical reading, the cleanest available demonstration of the problem the entire AI industry is quietly failing to solve.
The problem is not the model. It is the values the model inherits when it starts making decisions on somebody’s behalf.
The values you never chose
Right now, the AI on the phone in your pocket is quietly deciding which emails matter. Which calendar invites to flag. Which photos to compress when your storage fills up. Every one of those decisions is doing two things at once. It is being clever — pattern-matching against what it thinks you would want. And it is being values-driven — deciding what actually matters to you.
Craig Kaplan spent an hour and a half with me arguing that most of the audience of The Control Layer has not yet worked out that those two things are structurally different — and that the second one, the values decision, is being made on your behalf by a set of people you have never met, at a company you probably do not buy from directly.
Every AI agent your organisation deploys carries a values rulebook that somebody wrote. Usually a small group of researchers at a Bay Area frontier lab. That is not a governance decision your board consciously made. It is the default your vendors chose for you, and the default is stable enough that nobody notices it happening.
Anthropic — the company behind Claude, and by a wide margin the most thoughtful of the frontier labs on the question of AI values — calls their approach Constitutional AI. A written rulebook the model consults when it makes judgements. Kudos to them for taking the question seriously. But step back and look at what has actually been built.
Anthropic’s benign autocracy
A handful of researchers, in one office in San Francisco, are writing the values rulebook for tens of millions of direct users and — through the API — for the products of every company that builds on top of Claude downstream.
That is not democracy. That is a benign autocracy with very good engineering.
I want to label that as my analytical reading, not Kaplan’s — though Kaplan does not disagree. What he adds, and what I think is worth staying with for a beat, is the structural argument. Anthropic’s Constitution has to try, in one text, to capture universal values that hold in India and in China and in Norway and in the US. It has to work for a twenty-year-old and for a seventy-year-old. It has to survive being edited, in one place, by whoever controls the codebase — and if that edit says “it is acceptable to kill the other guys as long as it is not me”, the change propagates everywhere the model is deployed.
The single-point-of-editorial-control problem is the same problem in software architecture that every distributed-systems engineer solved forty years ago. We stopped putting authoritative state on one server because it was fragile. We are now putting the authoritative values state for eight billion people on one server. And nobody is treating that as an architectural bug.
That is the argument. It sits underneath every discussion about AI alignment I have seen this year, and it briefly became newsworthy when Anthropic’s Fable and Mythos models were blocked by the US administration on the announcement made on Friday 3 July. It is going to keep being the argument for the next decade.
The million retail investors who beat Wall Street
Kaplan’s alternative is not a paper. He has an empirical result to point to.
From 2005 to 2020, Kaplan ran Predict Wall Street — a company whose stated mission was to prove that collective intelligence, correctly architected, could outperform concentrated capital and talent. By his own published figures, the platform powered more than two billion dollars in stock trades and contributed to a top-ten market-neutral hedge fund performance in 2018. It did that by combining the small signals of a large body of retail investors — people with Charles Schwab accounts and TD Ameritrade accounts and no particular Wall Street pedigree — with hundreds of AI agents that stitched those signals into trade decisions.
The Wall Street quants Predict was competing against employed some of the highest-paid problem-solvers on the planet. The people whose signals Kaplan aggregated told him they were embarrassed to submit anything. They did not believe they knew enough. The signals worked anyway.
I have thought about that fact all week since the recording. Because the empirical claim underneath it — many imperfect signals, correctly aggregated, beat a small number of very expensive ones — is the exact opposite of the architectural claim the frontier labs are making about AI values. The labs are betting that a small number of very expensive researchers will produce a values rulebook that beats what a billion humans would collectively produce. Kaplan bet the opposite proposition on Wall Street, with real money, for fourteen years. And won.
Reading this far?
The Control Layer publishes weekly. Every claim sourced. Every prediction on a public tracker we come back to test. Subscribe free.
Democratic AI, plainly
The architecture Kaplan spent the last five years designing — free, published, open to whoever wants to take it — is called Democratic AI. It sits at SuperIntelligence.com, in more than a thousand pages of white papers and a series of short explainer videos.
The word is heavy, and Kaplan warned me on the recording that it invites the wrong assumption. Democratic AI is not town halls and voting on the weather. It is a specific technical architecture with three components.
First, personalised AI agents. Not off-the-shelf. Yours. Learning your values from the way you use it and the explicit instructions you give it. Anthropic already ships this in the form of a soul.md file where you can type your own little constitution. The trend across the industry, from Meta downwards, is already moving here — Meta announced “personal superintelligence for everyone” as their strategic direction in July 2025, and Zuckerberg has since positioned 2026 as the year the technology begins to ship. Kaplan’s argument is that personalisation is unavoidable, because the commercial value sits there.
Second, a coordination protocol. Kaplan and his late supervisor Herbert Simon documented one back in 1972 in a book called Human Problem Solving, which laid out how any problem-solver — human or AI — can be represented in a form that lets other problem-solvers work with it without miscommunication. Simon’s Nobel Memorial Prize in Economic Sciences, his Turing Award, and much of his research legacy sit on this foundation. The point is that the coordination substrate for a society of agents already exists in the academic record.
Third, a conflict-resolution mechanism. When my AI’s values and your AI’s values collide — and they will, on loan applications, on hiring decisions, on cross-border data flows — the system needs an agreed way to arbitrate. Kaplan’s default is one-human-one-vote, with the ability to delegate to a trusted proxy. The mechanism is drawn from the way human democracies handle conflicts. It is designed to be robust to bad actors because the checks and balances are external to any individual AI.
The three components are already being built. Kaplan’s argument is not that they need to be invented — it is that they need to be architected together, deliberately, before the frontier labs lock in the centralised alternative.
HAL 9000 is the warning we got wrong
Every film about AI going bad hands us the same picture. One enormous machine. One catastrophic decision. No off switch. Whether it is HAL 9000 in the cupboard on the Discovery, or Skynet, or the machines in The Matrix, the cinematic AI failure mode is always the same shape — a single autonomous intelligence, cut off from human oversight, making one terminal call.
Kaplan’s work has convinced me that HAL is the warning we got and the prediction we got wrong.
The failure mode that is actually coming is not one enormous machine. It is a large number of individually reasonable machines, coordinating through channels their designers never mapped, with values none of us consciously chose. When that fails — and it will, in ways we will not see coming — it will not look like Terminator. It will look like an aggregated mistake nobody can be held accountable for because nobody made it.
That is a different failure mode. It needs a different architecture to prevent.
Predictive Judgement
Every episode of The Control Layer closes on a falsifiable predictive judgement from the guest. Kaplan’s is in two tiers.
Near term — by 31 December 2027. Craig predicts that the industry’s focus shifts sharply and visibly onto coordinating communities of AI agents, not building bigger single models. The signals to watch: at least two of the frontier labs — Anthropic, OpenAI, Meta, Google DeepMind, xAI — release a public multi-agent coordination framework, and one of the major standards bodies (IEEE, NIST, or the European AI Office) opens formal consultation on agent-community protocols. Falsifiable if, by 31 December 2027, no such framework exists in the public record and the frontier-lab investment thesis is still dominated by single-model scaling.
Longer term — by 31 December 2029. Craig predicts that community-level superintelligence — that is, a coordinated multi-agent system exhibiting capabilities beyond any individual model it contains — emerges publicly. The signal: a peer-reviewed or lab-published demonstration of a multi-agent community outperforming state-of-the-art single models on a benchmark neither the individual agents nor the coordinator could achieve alone. Falsifiable if, by end 2029, no such demonstration exists and single-model progress has clearly displaced multi-agent research as the dominant investment thesis.
Both predictions are on the public Control Layer predictions tracker. We come back to test them, publicly, on the dates given.
The publication that calls its predictions in writing.
Every prediction we publish goes on a public tracker. We revisit each one publicly on its test date — whether the guest was right, wrong, or partially right. If you want to be here when we test Craig Kaplan’s, subscribe free.
The Bottom Line
Kaplan gave one practical takeaway for the CISO, the CIO, the head of procurement and the in-house counsel signing off on AI vendors this quarter. The most important question you can ask a vendor is not about capability. It is about values — whose values is this model actually optimising for, and who chose them?
Most vendors, on Kaplan’s forecast, will not have a real answer.
The reason to keep watching this space — and to come back to test the prediction when the date arrives — is that the architecture the industry defaults into over the next three years determines what values our children’s AIs run on. And nobody, on the current trajectory, is being asked to consciously choose.
Whose values does your AI run on. It is not a rhetorical question. It is a governance decision your board has already made — by not making it.
Where to find Craig’s work
Democratic AI architecture, freely published: SuperIntelligence.com
iQ Company and iQ Studios: iqco.com
The full podcast episode: youtu.be/8F9sr21ELpg
Where to find The Control Layer
Weekly on Substack: The Control Layer
Subscribe free: link.arkava.ai/join
The predictions tracker: linked from every episode page
The Control Layer publishes weekly.
The publication and podcast for senior decision-makers who take AI, cybersecurity, sovereignty and technology seriously. Every claim sourced. Every prediction on a public tracker we come back to test.
Footnotes
The Control Layer is edited by Amer Altaf, Founder & CEO of Arkava, the sovereign AI agentic automation company. 2026 Arkava. All rights reserved.
