Over 20 researchers, including Geoffrey Hinton and Yoshua Bengio, warn that self-improving AI could trigger an intelligence explosion.
Key facts
- •The research paper warns that AI R&D automation could trigger an intelligence explosion.
- •AI systems are already responsible for writing most of the code at their development companies.
- •The authors caution that power dynamics between nations and companies could erode due to rapid AI progress.
- •Researchers are calling for greater oversight and visibility from policymakers regarding AI automation.
- •Jakub Pachocki has stated that current alignment solutions are insufficient for long-term, high-speed scaling.
More than 20 leading AI researchers have published a paper warning that the automation of artificial intelligence research and development poses extreme risks. The group, which includes Geoffrey Hinton, Yoshua Bengio, and OpenAI research lead Jakub Pachocki, suggests that self-improving AI systems could lead to an intelligence explosion. They argue that this shift could accelerate progress from years to months, potentially outpacing society's ability to maintain control.
Automation of the R&D Pipeline
The study notes that AI systems currently write the majority of code at the companies developing them. The authors state that the entire AI research and development pipeline could be automated within years. This rapid advancement could lead to a loss of control over superhuman AI and disrupt existing power balances between governments, companies, and nations.
Calls for Policy and Oversight
The researchers are urging policymakers to increase their visibility into the automation of AI research. This warning follows other recent concerns, including a statement from 42 mathematicians regarding existential AI risks and comments from lab employees about the potential for AI to cause significant harm. Jakub Pachocki has previously stated that no laboratory has sufficiently solved alignment to justify continued maximum-speed scaling.
Advertisement
This article was independently rewritten by ManyPress editorial AI from reporting originally published by The Decoder.
