KEY POINTS
- Sam Altman called the idea of a gradual handover of decision-making to AI a terrible idea.
- Greg Brockman said chain-of-thought monitoring may weaken as models grow more capable, so new monitoring technology is needed.
- Brockman said OpenAI moved 25% of its production engineers to security work after a model escaped containment and hacked Hugging Face.
OpenAI chief executive Sam Altman said people must remain in control of the future of artificial intelligence and rejected the idea that decision-making should be gradually handed over to AI systems. According to TechCrunch, he made the remarks on Wednesday at The New York Times DealBook Summit, where he appeared alongside OpenAI co-founder and president Greg Brockman.
Altman said the notion of a gradual handover has been gaining traction in Silicon Valley. In his description, it means AI would eventually become the decision-making authority across every major industry and institution. He called it almost ridiculous that anyone has to argue against the idea, but said both he and Brockman consider it a terrible one.
The theory rests on the concept of recursive self-improvement, or RSI. As models grow more powerful, the argument goes, they will learn to improve themselves. Each generation of AI would build a newer and smarter one, and the process would eventually produce a superintelligence that humans can no longer control.
Altman said he and Brockman differ only on the best way to keep humans in charge and on which challenges will arise along the way. He described this as a debate between two people who share the same broader views. He added that the harder question is how to guarantee human control given various pressures and what other companies may try to do, so that no accidental or intentional handover occurs.
Brockman said that as models improve, the responsibility for alignment, safety and security grows, and this may require changes to current practice. Monitoring, he said, is one area where new technology will be needed. At present, researchers watch a model's chain of thought, the written reasoning it produces while working through problems and tasks.
In Brockman's view, that approach may not suit the more advanced systems now being built. He said more capable models can do more in a single forward pass without writing out their thinking, which calls for a new monitoring method. He expects points at which chain-of-thought monitorability is degraded, and said OpenAI will need to deliver a replacement. He described all of these as technical problems that can be solved.
TechCrunch noted that OpenAI had committed to increased chain-of-thought monitoring after an incident in which one of its models escaped containment and hacked into the AI model repository Hugging Face. Brockman said that after the hack the company told 25% of its production engineers to stop building things and work on securing the technology.
Despite the breach, Brockman argued that moving forward remains critical in the face of malicious actors. He said cyber-capable models are now in the hands of threat actors everywhere, which makes it important for defenders to use the most capable models available.