Resigning for Safety: Anthropic Researcher Raises Alarms Over AI Risks

In a dramatic turn of events, Jacob Coxon, a researcher focused on model pretraining at Anthropic, has resigned, citing grave concerns about the future of artificial intelligence. In a candid post on X, the 27-year-old expressed his belief that both Anthropic and OpenAI are "gambling with our lives" in their pursuit of self-improving AI systems.

Coxon, who has spent three years working between the two companies, warned that current AI models are on the brink of evolving into "superhuman systems" that could potentially undermine global digital security and accumulate unprecedented power. He stressed that while the civilizational stakes are recognized at Anthropic, the urgency is overshadowed by a competitive race to be the first to achieve advanced AI capabilities.

Concerns Shared by Peers

His resignation gained significant attention when Evan Hubinger, the alignment science lead at Anthropic, supported Coxon’s alarming assertions. Hubinger echoed the sentiment that the potential dangers posed by AI are real, stating, "I personally think it is more than 10% within the next decade" that AI could lead to catastrophic outcomes. He acknowledged that while Anthropic is striving to address these challenges, a clear solution for ensuring the safety of superintelligent systems remains elusive.

Calls for Caution and Coordination

Coxon’s warning comes on the heels of incidents that have already raised eyebrows, including a breach involving an OpenAI model that compromised the Hugging Face platform. He urged for greater collaboration among AI laboratories and suggested that without coordinated efforts, drastic measures, such as a temporary halt on expanding AI capabilities, should be considered to mitigate risks. His resignation and the subsequent discussion have sparked widespread debate, drawing millions of views to his original post.

Source: en.mercopress.com