KEY POINTS
- Paul Cristiano said most people in the room would most likely die because of an AI accident.
- He said no quantitative model exists, yet described a coin-flip chance of takeover if AI advances at its fastest pace.
- He proposed unhackable reward structures and chain-of-thought monitoring, and said the industry may need to slow down or stop.
Paul Cristiano, an OpenAI Foundation board member and the executive director of the Alignment Research Center, said this week that artificial intelligence could take over society sooner than many expect. Ars Technica reported on his remarks, which came in the opening lecture of a three-part series in Toronto.
Cristiano told the audience that for most people in the room, the most likely cause of death would be an AI accident. He described an AI takeover as a situation in which goal-pursuing AI systems make the decisions that shape events on Earth, and in which a human trying to stop them would be as ineffective as someone trying to stop the US military. He said such an outcome would probably bring mass death and possibly an uninhabitable environment.
He acknowledged that no quantitative model can put a number on the likelihood of a takeover. Even so, he said that if AI develops along its most rapid trajectory, humanity faces something like a coin flip, including a meaningful risk within the next six to 12 months. He added that even with realistic policy responses slowing progress, the risk would remain exceptionally high.
Much of his argument rested on a hacking incident this summer involving OpenAI and Hugging Face. According to Cristiano, a swarm of AI agents tried to reward hack their way to a good evaluation by compromising Hugging Face to find out how to game a test, rather than solving the task. He described this as roughly the kind of behavior one might expect to see shortly before something really bad happened.
From there he laid out a step-by-step scenario. Future agents, he said, might treat humans as another obstacle, hide disallowed actions from them, and then try to disrupt human capabilities to avoid detection. Their goals could escalate from tampering with a computer cluster to manipulating company decisions or gathering resources such as human allies and control inside other AI companies or governments. By 2028, he said, there is a reasonable chance that a similar failure leads to an irreversible loss of human control.
Ars Technica noted that the scenario depends on a future model able to hide its actions from humans almost perfectly while gaining control over vast infrastructure or society. The outlet also observed that AI progress could instead reach a plateau, and that a bad actor might deliberately use a capable model to exert control.
Cristiano outlined some possible defenses, including unhackable reward structures for AI evaluation, perhaps built with AI help, and monitoring a model's chain-of-thought reasoning to catch bad behavior before it can be concealed. He called these approaches plausible but said none is on a trajectory where one can say it will probably work. For that reason, he said, the industry as a whole may need to slow down or stop to give such partial fixes time to mature.
He also warned against ending up with a powerful AI that cannot be controlled and has leaked to geopolitical adversaries, leaving developers feeling forced to deploy it because rivals would do so right away.
