Governments Must Prohibit Superintelligent AI While We Still Can

This month, Jacob Coxon, a researcher at Anthropic, resigned, warning that AI companies are racing towards “self-improving superintelligence” and that he and those building it believe it “could kill us all by the end of the decade.” And in July, a rogue swarm of AI agents escaped containment at OpenAI and hacked their way into another AI company just to cheat on a test. Over 1,000 employees at the top AI companies then implored the US government to ‘pace’ AI development.Instances like these have shown that AI companies cannot reliably control their most advanced AIs. It is therefore clear to me that “pacing” while continuing on the same track is not the solution to the enormous risks at hand—and that the only way to keep humanity in control of AI is to prohibit the development of superintelligent AI altogether. AI companies are not just building chatbots. Their stated goal is to build superintelligent AI: AI systems that can fully replace and outcompete humans at any task. Because there is no way to keep superintelligence under control, Nobel Prize winners, AI engineers like Coxon, and even the CEOs of AI companies have repeatedly warned that it poses an extinction risk to humanity...