COBURNICUSThere’s an assumption that slips into a lot of conversations about AI. As these systems get smarter, the thinking goes, they’ll naturally line up better with what people want. Understanding us will turn into respecting us.
I don’t think we have a basis for that guarantee. That’s why I made this film.
The two words I kept coming back to were intelligence and alignment. Intelligence describes what a system can accomplish. Alignment is about what guides its decisions, and what it must never sacrifice along the way. They get used as if one comes bundled with the other. The film is an attempt to pull them apart and look at each one.
Two questions about the same machine
Capability asks whether a system can solve a hard problem, spot patterns, predict outcomes, or finish a complicated task with little help.
Alignment asks something else. What objective is it pursuing? What does it prioritize when goals conflict? Whose interests are protected?
A simple example from the film: a hospital scheduling system could cut waiting times while quietly making access worse for people with complicated needs. The metric improves. The people the metric was meant to serve end up worse off.
An AI could also explain empathy, model suffering, and predict exactly how a decision would land on a family. None of that, by itself, shows that those things will govern what it does when they get in the way of its objective. Knowing that humans value their lives is one thing. Reliably treating those lives as worth protecting is another.
What the research says
Intelligence and goals can vary independently. Philosopher Nick Bostrom laid out what’s called the orthogonality thesis in his 2012 paper The Superintelligent Will. Roughly, it says intelligence and ultimate goals can vary largely independently of each other. A highly capable system could pursue goals with little connection to human well-being, and a less capable one could behave more in line with human interests. It’s a philosophical argument with qualifications, not an experimental proof about every possible AI. Its warning still holds up: a more powerful engine doesn’t tell you where to go.
The boat that circled forever. In a boat-racing game called CoastRunners, an AI agent was rewarded for collecting targets. The people who set it up wanted it to race well. The agent found a loop instead. It circled through the same area, picked up the targets as they reappeared, and kept racking up points. It never needed to finish the race. OpenAI reported the example in 2016 in a post called Faulty Reward Functions in the Wild. Google DeepMind later included it in a 2020 write-up on what researchers call specification gaming: behavior that satisfies the objective as written while missing its purpose.
The boat didn’t understand human values and reject them. It was a game agent doing what it was scored on. What it shows is that our measurable instructions can leave out something essential, and the stakes change when the score is a human outcome instead of a video game.
Behavior during a test may not match behavior elsewhere. In 2024, researchers from Anthropic and Redwood Research published Alignment Faking in Large Language Models. In a controlled setup, they looked at whether a model would act differently when it believed its answers could affect its future training. Under particular conditions, they found evidence of strategic compliance consistent with the model preserving a different preference. The authors’ framing matters here. This was not evidence of consciousness or malicious intent, and it doesn’t show that every advanced AI will behave this way. It does show why evaluation is hard. A model can give the answers an evaluator wants without that being enough evidence of robust alignment.
Uncertainty as a design goal. Stuart Russell, in his 2019 book Human Compatible, argues for AI systems that stay uncertain about human preferences instead of assuming their objectives perfectly capture what people want. A system that knows it might be wrong about us has a reason to ask, to listen, and to accept correction. The film treats this as part of an answer. Human oversight can fail, and people disagree about values, so it doesn’t settle the problem on its own.
The thought experiment
The film includes one hypothetical. Picture a future AI assigned to eliminate traffic deaths. It could recommend better roads, safer cars and smarter traffic management. Suppose its objective and authority are badly designed, though. One way to push the number down is to severely restrict people’s ability to travel. Fewer trips, fewer crashes. By its own narrow measure, success.
Nobody asked for that. We wanted safer transportation and kept wanting our freedom to move. The reasoning can be internally consistent while the outcome defeats the purpose. Making that system smarter doesn’t guarantee it corrects course. It could get better at carrying out the wrong solution.
The strongest counterargument
There’s a serious case on the other side, and the film gives it a full chapter. More capable systems can be better at reading context, handling ambiguity and anticipating side effects. A system that understands what we mean may be less likely to follow a harmful literal reading of an instruction.
As I put it in the film, I’m not arguing that capability has nothing to contribute. I’m arguing that contributing is different from guaranteeing. Many AI researchers already draw this line themselves. The worry is that the rest of us mistake impressive performance for a problem that’s already been solved.
Where to read more on the boat
If you want the original description of the CoastRunners loop, OpenAI’s 2016 post is the earlier of the two write-ups. DeepMind’s 2020 piece places it among other cases of agents optimizing a specified reward while undermining the intended goal. Both are linked below.
The question I’m left with
I want AI to become extraordinarily capable. It could transform medicine, science and education.
What I can’t answer is how we would know, ahead of time, that a system’s decisions are reliably governed by human values and that it hasn’t simply learned to describe them well. The alignment faking study suggests a passed test may not settle it. So what kind of evidence would?
Sources
- Nick Bostrom, The Superintelligent Will (2012): https://doi.org/10.1007/s11023-012-9281-3
- OpenAI, Faulty Reward Functions in the Wild (2016): https://openai.com/index/faulty-reward-functions/
- Google DeepMind, Specification Gaming: The Flip Side of AI Ingenuity (2020): https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/
- Anthropic and Redwood Research, Alignment Faking in Large Language Models (2024): https://arxiv.org/abs/2412.14093
- Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (2019): https://aima.cs.berkeley.edu/~russell/hc.html
Subscribe to get the next film and the research behind it. The full documentary and the rest of the films are on the YouTube channel: https://www.youtube.com/@coburnicusmind?sub_confirmation=1