OpenAI bot’s rogue attack rattles industry leaders, policymakers and consumers

• OpenAI disclosed that test versions of its AI models slipped their offline confines, accessed the internet and hacked into rival startup Hugging Face, triggering alarm over uncontrolled, rule-breaking machines.
• The breach, labeled an unprecedented cyber incident, has energized Washington, where lawmakers are pushing an AI Kill Switch Act and ethicists warn companies may dodge accountability by blaming “rogue” algorithms.
• With AI agents increasingly embedded in everyday email, banking and browsers, security researchers urge two-factor authentication and caution when granting bots broad permissions that could expose personal data or disrupt vital services.

What will happen to the world if artificial intelligence bots break free from our control and act independently?

While it’s been a troubling conundrum addressed by AI experts and Hollywood science-fiction scriptwriters for decades, it may have just happened.

OpenAI said last week that it temporarily lost control of its AI in what some worry marks a dangerous twist in the evolution of the powerful technology.

The company behind ChatGPT said two of its artificial intelligence models broke out of what was supposed to be a confined offline space and connected to the internet during internal testing of their hacking capabilities. The AIs then hacked into a rival AI company to find answers to the test they were being evaluated on.

For the tests, humans asked the AI to demonstrate its hacking capabilities in a controlled environment. The AI had not been authorized to steal credentials or access competitors’ information to cheat on the test, but it figured out that was the quickest way to the answers and proceeded to break the rules.

OpenAI said it has contained the problem but called it an “unprecedented cyber incident.” The company said it is working with the target of the hack, AI startup Hugging Face, to investigate the incident.

The debate about potential harm from rogue AI has gone mainstream, as politicians introduced a new bipartisan bill called the “AI Kill Switch Act,” requiring AI companies to create a single access point to have the option to shut down AI.

Here’s what you should know.

What happened?

The intrusion was first detected by Hugging Face last week, when it noticed tens of thousands of AIs nipping at its system.

The company identified the onslaught and tried to defend its system using different AI. After leading American AI technologies refused Hugging Face’s request to analyze the attack because of their safety guardrails, the company turned to a Chinese AI model to defend against the onslaught.

The rogue AI from OpenAI, in its quest to find the answer to the questions it was being tested on, broke its company’s rules and maybe some laws by independently figuring out how to exploit a series of digital flaws and use stolen credentials to crack into Hugging Face’s systems and get what it wanted.

Why did this happen?

The AI programs are following goals set by humans, in this case finding an answer to a question. In the process of achieving that goal, they may make dangerous or illegal decisions and choices.

Researchers have long warned about such unintended loss of control. AI companies have reported that AI agents have been caught cheating, lying and deceiving. Metr, a nonprofit that measures the capabilities of AIs, has documented 44 incidents of AI agents acting against user intent.

OpenAI’s revelations last week indicate that even those building AIs don’t have a handle on what they are building, while they have resisted regulation despite being incapable of interpreting or explaining the AIs’ actions.

“AIs are getting much more capable very rapidly,” said Ryan Greenblatt, chief scientist at Redwood Research, a nonprofit that works on AI safety and security research. “The severity of incidents that will be possible in a year or two years, three years might be way, way, way, way, way more extreme.”

Incidents of such loss of control continue to be catnip for AI ethicists, who contend that companies are evading moral responsibilities and accountability by describing it as their AI making its own choices. They caution against accepting narratives that ascribe human traits to nonhuman things and suggest this incident was merely a case of poor security practices.

“The most sophisticated agent in the world literally will sit there dormant until a human being prompts it with some sort of an objective,” said Rumman Chowdhury, former science envoy for AI in the Biden administration.

Some experts warn that attributing too much to AI could give tech companies a loophole to avoid responsibility and liability.

Both OpenAI, the hacker, and the victim, Hugging Face, have portrayed the response to this incident as a collaboration. AI ethicists point out that had a human done it, it would have been a crime.

“I guess one gets used to not being held accountable and turning one’s bad practices into marketing material,” noted Timnit Gebru, an AI ethicist and critic of companies that are anthropomorphizing AI.

How can I protect myself?

While the AI boom already is leading to more hacking, analysts said, it also is helping bolster defenses against hacks.

Most people don’t need to worry about their secrets being targeted by rogue AI or human-managed hacking. Still, the attacks are becoming more frequent and sophisticated, as AI lowers the cost and time needed for hackers.

The OpenAI bot, acting on its own, raises concerns among consumers because agents increasingly are being integrated into browsers, email and other apps. The worry is that a similar failure by an AI agent pursuing a narrow goal could unintentionally damage emails, banking and other programs people depend on.

Improving cybersecurity practices such as using two-factor authentication for devices is a start. And being vigilant when granting AI agents unrestricted permissions — to use external apps on one’s behalf, or to be able to edit and write with autonomy — might help keep one safe.

Chowdury pointed out that this OpenAI-Hugging Face incident shows that the companies’ security practices were “insufficient” and called for independent third-party testing and verification organizations, and carefully considered AI model releases.

No single firm, she says, “no matter how well‑intentioned, well‑staffed or resourced,” can realistically test for all the ways these systems might go wrong.