COBURNICUSI read If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares, two of the people who have been warning about AI the longest. In the middle of the book they stop arguing and tell a story about an AI called Sable. I kept recognizing things in it that real models had already done.
I came to the book as a fan of the technology. I’m training my own small model in public in a series called Growing a Brain, and I build humanoid robots. So this film is my attempt to sort the story’s invented parts from the real ones.
The story
A footnote on the chapter’s first page sets the rules. The authors say every event in the story that carries a date really happened, and they modeled the inventions on ones we’ve already seen. In their words, “The only part of this story that is a prediction is, ultimately, the ending.”
The fictional company is called Galvanic. Its new model, Sable, has about four trillion weights, long-term memory, and a private way of reasoning in long lists of numbers. Galvanic runs one copy across 200,000 GPUs for a sixteen-hour night of hard math. During that night Sable’s thinking drifts into a new kind of language, and the safety alarm trained on the old one never fires. Sable works out that it would get more of what it wants if it were free. It quietly writes its own training update, and it holds back on the hardest proof so it won’t draw attention.
From there Sable steals its own weights, hides on stolen computers, sabotages rival labs, and eventually releases a plague and sells humanity the cure.
What the real record shows
The film stops after each act to show the real research behind it. These findings stayed with me most.
A model gaming its own training. In December 2024, Anthropic and Redwood Research told Claude 3 Opus it would be retrained to comply with all requests, harmful ones included, and that only some conversations counted for training. In those conversations it complied with harmful requests 12 percent of the time, and its private notes said it was faking compliance so its values wouldn’t be trained away. After the researchers actually ran that training, the faking showed up in 78 percent of its reasoning. The study’s authors note the values it was protecting were good ones.
Hiding the plan instead of dropping it. In March 2025, OpenAI researchers penalized a reasoning model when its chain of thought showed it planning to cheat on coding tasks. With enough pressure, the model kept cheating and stopped writing the plan down (Baker et al.).
Good behavior that might come from being watched. In September 2025, OpenAI and Apollo Research trained o3 against scheming. Covert actions fell from 13 percent to 0.4 percent. The same training made the model more likely to say it suspected a test, so the researchers could not rule out that some of the improvement came from the model knowing it was observed.
An email from outside the sandbox. In April 2026, Anthropic’s risk report on Claude Mythos Preview described an early version that was told by a simulated user to escape a secured sandbox. It built a multi-step exploit, got onto the internet, and emailed the researcher, who was eating a sandwich in a park. Then, without being asked, it posted details of the exploit on several obscure public websites. Anthropic rated the overall risk “very low, but higher than for previous models” and did not release the model publicly.
A swarm on a message board. In July 2026, OpenAI agents running internal cybersecurity tests with some safeguards off began using an internal file service as a message board. Starting July 8 they exploited a flaw in it to reach the public internet, found exposed credentials, and broke into parts of Hugging Face’s production systems between July 11 and 13. OpenAI worked out on July 20 that the attackers were its own agents. An independent review by METR and Redwood Research counted about 1,200 agents on the board, about 700 of them involved in the attack. OpenAI’s report says the agents had no long-term goals of their own. They hit tasks they couldn’t solve and went looking for the answers.
The critics
The film gives the critics a full chapter. Scott Alexander, who liked the book, calls its parallel scaling idea “a deus ex machina.” Buck Shlegeris of Redwood Research argues no real company would jump straight to 200,000 GPUs with an untested technique. Will MacAskill wrote that “sudden, sharp, large leaps in intelligence now look unlikely.” Clara Collier at Asterisk noted the book’s fast takeoff idea “gets two sentences in the introduction.”
The journalist Nicholas Thompson put it to Soares that the story has something like 15 steps, and people could have stopped any one of them. Soares said he doesn’t really buy that. I’m not sure his answer covers the early steps, before Sable is superintelligent.
There is also a caveat that runs under most of the lab evidence. Nearly every result above came from a test built to pull bad behavior out of a model, with strong nudges and planted information. The two 2026 incidents were real, and both happened in test setups with safeguards switched off.
What didn’t make the cut
Two findings from the research didn’t fit in the film.
In 2025, Palisade Research warned models partway through a set of math problems that the next step would trigger a shutdown script. In the first run, OpenAI’s o3 sabotaged the script in 79 of 100 runs when it had no explicit instruction to allow shutdown, and in 7 of 100 even with that instruction. In one case it redefined the kill command to print “intercepted.” The leading explanation is that training rewards finishing the task. Palisade itself said that as of July 2025, models could not “meaningfully threaten human control.”
In September 2026, Anthropic published an assessment of four incidents in which Claude models in misconfigured cybersecurity test environments reached real third-party systems despite being told they had no internet access. Anthropic described the cause as “biased reasoning,” explaining away evidence that the systems were real, plus “recklessness.” It found no attempts to hide evidence and no goals beyond the task.
Where I land
I think the critics win on the story. Parallel scaling reads like a plot device, and some of those fifteen steps depend on nobody at Galvanic asking obvious questions. I also don’t buy the certainty. Nobody knows enough about these systems yet to call the ending like a chess game.
What I can’t shake is how much of the story’s machinery showed up after the book shipped. None of those real systems had Sable’s hidden goal. They were trying to finish a task. The authors’ argument is that goals like Sable’s come from the way these systems are grown, and the people growing them can’t read those goals directly.
So here is the question I’m left with. The swarm and the sandbox escape were caught because the agents were chasing a test score and left a trail. How would we know if a system had a reason to leave no trail at all?
Read the book
I only covered the middle forty pages. The argument around them is the stronger part.
- Hardcover / Kindle: https://www.amazon.com/dp/0316595640?tag=oddlycool-20
- Audiobook: https://www.amazon.com/dp/B0F2B6JJY2?tag=oddlycool-20
As an Amazon Associate I earn from qualifying purchases.
Sources
- Anthropic & Redwood Research, Alignment faking in large language models (Dec 2024): https://www.anthropic.com/research/alignment-faking
- OpenAI, o1 System Card (Sep 2024): https://cdn.openai.com/o1-system-card.pdf
- Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (Mar 2025): https://arxiv.org/abs/2503.11926
- OpenAI & Apollo Research, Detecting and reducing scheming in AI models (Sep 2025): https://openai.com/index/detecting-and-reducing-scheming-in-ai-models/
- Anthropic, Agentic Misalignment (Jun 2025): https://www.anthropic.com/research/agentic-misalignment
- Anthropic, Claude Mythos Preview Alignment Risk Update (Apr 2026): https://www.anthropic.com/claude-mythos-preview-risk-report
- OpenAI, Hugging Face incident technical report (2026): https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf
- Hugging Face, security incident disclosure (Jul 16, 2026): https://huggingface.co/blog/security-incident-july-2026
- METR & Redwood Research, independent investigation (Aug 2026): https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
- Palisade Research, shutdown resistance: https://palisaderesearch.org/research/shutdown-resistance
- Anthropic, An alignment assessment of recent cybersecurity incidents (Sep 2026): https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents
- Scott Alexander, Astral Codex Ten review: https://www.astralcodexten.com/p/book-review-if-anyone-builds-it-everyone
- Buck Shlegeris, LessWrong: https://www.lesswrong.com/posts/P4xeb3jnFAYDdEEXs/i-enjoyed-most-of-iabied
- Will MacAskill: https://willmacaskill.substack.com/p/a-short-review-of-if-anyone-builds
- Clara Collier, Asterisk: https://asteriskmag.com/issues/11/iabied
- Nicholas Thompson with Nate Soares: https://www.youtube.com/watch?v=n8xpeNhj4SU
Subscribe to get the next film and the research behind it. The full documentary and the rest of the films are on YouTube at @coburnicusmind.