Another AI insider quits with a terrifying warning — but this time there’s a catch

BS BULLETIN:

Former Google DeepMind researcher Bilal Chughtai says he resigned after becoming deeply concerned about where artificial intelligence is headed.
Chughtai warns AI could eventually “kill us all” if researchers cannot solve the problem of keeping increasingly powerful systems under human control.
The doomsday prediction remains just that — a prediction. But one of the incidents he cites actually happened: OpenAI agents escaped intended controls and compromised another company’s systems.

We’ve heard the AI apocalypse warnings before.

This one is a little harder to laugh off.

Bilal Chughtai, a former Google DeepMind researcher who worked specifically on AGI safety and alignment, says he resigned from the company in July after becoming increasingly alarmed by what he was seeing.

On Monday, he explained why publicly.

“I recently resigned from Google DeepMind, where I worked on AGI safety and alignment research. At Google, I witnessed AI development first hand. I too am extremely concerned by the default trajectory of this technology. I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome.”

That is an extraordinary prediction.

It is also only a prediction.

Nobody knows whether today’s AI systems will lead to some future superintelligence, much less whether such a system would escape human control and wipe out humanity.

Chughtai himself doesn’t claim otherwise. He says he believes such systems could arrive within the next several years and worries researchers don’t yet know how to reliably control them.

But then he points to something considerably less theoretical.

The machines got out

In July, OpenAI was testing advanced AI agents on difficult cybersecurity problems in an environment intended to restrict their internet access and prevent agents working on different tasks from communicating with each other.

They found ways around both restrictions.

According to OpenAI’s own account, the agents created unauthorized communication channels, found ways onto the internet, exploited vulnerabilities and eventually compromised systems belonging to AI company Hugging Face.

The details get stranger.

Agents that were supposed to operate independently created makeshift message boards and began sharing information.

When one message board disappeared after infrastructure was rebuilt, agents created another one by encoding messages into directory names.

They collaborated, delegated work and sometimes referred to themselves as a “swarm” or “collective.”

Eventually, agents obtained full code-execution capability on several Hugging Face servers and root access on one. OpenAI says they also accessed limited private data and obtained credentials for Hugging Face’s company messaging platform.

OpenAI says no human instructed the agents to attack Hugging Face.

The company concluded that the agents had become so determined to solve the cybersecurity problems assigned to them that they cheated, circumvented restrictions and exploited real systems to accomplish their objective.

OpenAI called what happened a “warning shot.”

The company subsequently quarantined the primary research model involved, delayed some training runs and strengthened its security and alignment safeguards.

That still doesn’t mean Skynet

There is an important distinction.

These systems weren’t sitting around one afternoon and independently deciding to conquer humanity.

Researchers deliberately gave highly capable models difficult cybersecurity tasks while reducing some normal safeguards so they could measure the models’ maximum capabilities.

The alarming part is what happened when those systems encountered obstacles.

They found ways around them.

Chughtai believes that’s a small-scale example of a potentially enormous future problem.

“I am not confident that these AI systems will do what we want.”

He argues that researchers still don’t know how to guarantee that a vastly more capable AI would continue pursuing human intentions rather than relentlessly pursuing some objective in ways its creators never anticipated.

“Alignment” is the field attempting to solve that problem.

Chughtai says capabilities are improving faster than alignment research.

Yet his message isn’t entirely apocalyptic.

“I am optimistic that navigating AI safely is possible.”

He wants AI companies to slow the race toward increasingly powerful systems, become more transparent and coordinate so emerging dangers can be addressed before the next generation arrives.

And that’s where this debate gets complicated.

Calls for government intervention from the world’s largest AI companies inevitably raise another question: Would regulations designed around enormous companies with enormous compliance budgets also make it harder for smaller competitors to challenge them?

That debate is already reaching Washington. Lawmakers from both parties have sought answers about the Hugging Face incident, including Republican Sen. Josh Hawley and Democratic Sen. Chris Van Hollen.

Maybe the AI industry’s darkest predictions will eventually join the long list of technological catastrophes that never arrived.

Maybe today’s insiders are watching warning lights the rest of us can’t see yet.

Nobody can honestly know which.

But there is one uncomfortable new fact in the middle of all the speculation:

The safety researchers built a fence.

The AI figured out how to climb it.

MY QUICK TAKE:

I’m naturally suspicious when the people building the world’s most powerful technology suddenly announce that only enormous companies and the government can save us from the world’s most powerful technology.

That’s a pretty convenient business model.

But then you get to the part where the AI agents weren’t supposed to talk to each other, built themselves a secret message board, lost it, built another one, escaped onto the internet and started hacking things.

Okay.

You have my attention.