AI researcher quits Anthropic with a chilling warning: ‘It could kill us all’ by decade’s end

BS BULLETIN:

AI researcher Jacob Coxon resigned from Anthropic Tuesday after spending three years doing pretraining research at Anthropic and OpenAI, accusing both companies of racing toward self-improving superintelligence while “gambling with our lives.”
Coxon says people building advanced AI genuinely fear it could “kill us all by the end of the decade,” while warning future systems could become capable of hacking, rapidly advancing scientific fields and acquiring real-world power and resources.
Anthropic alignment researcher Evan Hubinger publicly backed the central warning, saying he personally puts the chance of AI killing all humans at greater than 10% within the next decade — while stressing that he considers the risk from today’s models low.

The people building some of the world’s most powerful artificial intelligence systems have a message for humanity.

This may not end well.

Jacob Coxon resigned from Anthropic this week after spending the past three years working on pretraining research at both Anthropic and OpenAI — and then publicly unloaded on the industry he was leaving.

“I resigned from Anthropic today,” Coxon announced Tuesday.

His reason was considerably more dramatic than a disagreement over vacation days.

Coxon accused Anthropic and OpenAI of racing toward “self-improving superintelligence” without adequate safeguards and said the companies are “gambling with our lives.”

Then came the sentence that sent his warning racing across the internet.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote.

According to Coxon, executives and senior researchers sometimes temper their language publicly while expressing much greater concern privately.

He argued that sufficiently advanced systems could eventually become capable of hacking computer systems, rapidly advancing entire scientific fields and obtaining resources and influence in the real world.

And Coxon isn’t the only Anthropic insider sounding the alarm.

Evan Hubinger, an Anthropic alignment researcher, responded publicly that Coxon was correct about researchers genuinely taking the possibility of human extinction seriously.

Hubinger said his own estimate of that risk is greater than 10% within the next decade and acknowledged that Anthropic does not yet have a solution for safely aligning a hypothetical superintelligence.

He subsequently added an important qualification: Hubinger believes the danger posed by present-day AI models is low. His concern is what could happen if future systems achieve superintelligence through rapid recursive self-improvement.

Coxon’s resignation has quickly become much bigger than one employee leaving one AI company. The Wall Street Journal, Axios, The Verge and other outlets picked up the story Wednesday as his warning spread online.

Coxon is calling for AI companies to coordinate rather than race one another and for governments to intervene if necessary to slow the development of systems that could become uncontrollable.

In other words, one of the guys who actually helped build this stuff has decided he’d rather get off the train.

And on his way out, he yelled back to the rest of us that the brakes may not work.