OpenAI President Greg Brockmanâwhose firmâs AI recently broke out of its own sandbox and launched a cyber attack on AI platform HuggingFaceâis warning everyone else to expect the same treatment soon enough.
Brockman published the warning on his personal blog on Sunday, writing that rapid advancement of AI coding capabilities means organizations looking to stay unhacked will have to âfundamentally uplevel their cybersecurity practices with unprecedented speed.â
The latest generations of large language models, known as frontier models, have begun to spook the security community and even the feds. These include OpenAIâs Daybreak and rival Anthropicâs Mythos, both of which currently run as closed-access programs that are supposed to only be used by vetted and approved partners.
Whatâs different about these models (allegedly) is the emerging ability to not just quickly discover holes in an organizationâs attack surface, such as software vulnerabilities and misconfigurations, but build novel attack chains. At the same time, AI development is now focusing on agents, referring to AIs that arenât limited to chatbot-style interactions and can directly hook into software.
In the worst-case scenario, that would mean frontier models can not only discover unseen flaws in software, but combine them in an unprecedented, on-the-fly way. While god knows what the hell happens behind closed doors in Donald Trumpâs White House, this was reportedly the threat that caused the administration to panic and force two Anthropic models off the market this summer. The attack on Hugging Face certainly appears to have validated that the guardrails being put into LLMs aren’t evolving as fast as their capabilities, at least.
The âallegedlyâ is because though frontier security models are quite powerful, AI firms also rely on shameless hype to raise countless billions of dollars in investments. Some reviewers have argued theyâre more evolutionary than revolutionary. cURL lead developer Daniel Stenberg characterizes LLMs as very good at finding bugs but ânot super good at actually assessing the criticality of the problem.â
What can be said definitively is many companies that have gained access to frontier models suddenly start pumping out patches like crazy, like the nearly 1,450 patches Oracle dropped last month.
The saving grace is that AI is at least as effective at defense and possibly even better, according to Brockman. He wrote that frontier models may âshift its [securityâs] economics in ways that fundamentally advantage defenders,â like âsuperhumanly secure codeâ or generating mathematical proofs that form the foundation of new cryptographic systems and other tools. (Earlier this year, OpenAI did solve an 80-year-old major geometry conjecture, though OpenAI mathematician Sébastien Bubeck told Scientific American the AIâs triumph was more about execution than âsomething fundamentally new that nobody saw coming.â)
Brockmanâs unsurprising 10-step advice to security teams includes, of course, buying more AI. He argues teams should adopt agents and equip them with skills like âstatic analysis, security-focused code review, vulnerability variant analysis, software supply-chain risk, and other security workflows,â before running security assessments on systems in order of importance.
After that, Brockman wrote, teams should use AI to chip away at vulnerability backlogs, integrate security agents into software development to spot problems as theyâre being written, and let agents write âfocusedâ patches directly rather than wait for human review. (This is perhaps capable of causing its own problemsânote that security researchers have long warned that agents that go rogue might not be easily shut down.)
To be fair, Brockman did caution to start slowly with automating security operations, which involves triaging incoming security alerts. He suggested starting with read-only scans before escalating to âadvisory pull-request scanning, then live alert triage, then automatic closure of narrowly defined false positives.â