Anthropic generated a lot of attention earlier this month when it
announced Claude Mythos Preview, a new frontier model that the major AI vendor said was exceptionally good not only at detecting and identifying software vulnerabilities – some even decades old – but also at developing exploits for them.
It was so good at exploiting the security flaws that Anthropic opted to limit Mythos’ release to certain vendors and researchers to allow them to develop better cyber defenses – as part of the AI company’s
Project Glasswing – rather than make it generally available and have it land in the hands of threat actors.
The news of Mythos’ cyber capabilities sent shockwaves through the tech industry and beyond, with some worrying that it would make protecting software even more difficult for governments and international bodies, sending warnings to banks and other financial institutions. There were also accusations that the extent of Mythos’ impact was being overhyped and possibly being put forward by Anthropic to boost its expected IPO.
Putting AI Models on Defense
Amid all of this, startup
Simbian, which offers an autonomous security operations center (SOC) platform to protect against AI-based attacks, tested 11 of the top AI frontier models to see how good they are at detecting MITRE ATT&CK sequences in complex and realistic scenarios – essentially, how well they can find attackers.
According to Simbian researchers, all of them failed. The large language models – which included Anthropic’s Claude Opus 4.6 and 4.7,
Google’s Gemini 3.1 Pro and 3 Flash,
OpenAI’s GPT 5, and
DeepSeek 3.2 – ran through 105 attack procedures in an agentic React loop against real attack telemetry. They were told to find both the attackers and their tactics.
Claude Opus 4.6 did best, averaging 46% success and finding three times more flags than Gemini 3 Flash, but at 100 times higher cost. A score of 50% was needed to pass. None of the models passed, and every one missed entire attack categories, Simbian’s newly launched research lab
wrote in a blog post.
Poor Results
The company summarized the results in its
Cyber Defense Benchmark report this week.
“Our research shows a few things,” Simbian co-founder and CEO
Ambuj Kumar told MSSP Alert. “Defense is very different from and harder than offense. The same models that excel at offense fail miserably at defense. During a cyberattack, you have a clear target, but during defense, you have partial trails of the attacker at best. You can throw in an LLM, and it'll find a vulnerability in an application. However, ask it to detect if someone else is exploiting the same vulnerability, and it may get confused.”
Kumar said the test results surprised him.
“These days, LLMs have gotten so good at almost everything,” he said. “Whether you ask them about general knowledge – the MMLU benchmark – math [in the] GSM8K benchmark, or anything else, they saturate the benchmark by scoring near 100%. Anthropic said Mythos does better than most humans on cyberattacks, therefore, it was shocking to see that even the best model could only score 4.49%” on Simbian’s benchmark.
Fighting AI with AI
AI – and in particular
agentic and reasoning AI – is changing the cybersecurity landscape, with both defenders and threat actors trying to use the fast-evolving technology to their advantage. Global investment firm
Morgan Stanley wrote that “AI is reshaping nearly every industry and cybersecurity is no exception,” noting that the technology’s ability to analyze massive datasets and find patterns can be a significant benefit to security teams and MSSPs.
However, Kumar said Simbian’s test is important because it’s the first benchmark to mimic real-world defenses, and the frontier models “failed miserably out of the box.”
“We have a conundrum,” he said. “On one hand, because of increased AI attacks, we must automate defenses with AI. But by default, the frontier models have severe limitations. The industry must know how to augment LLMs with outside intelligence – what we call ‘harness’ – to keep orgs secure.”
MSSPs Need a 'Data-Driven Approach'
For MSSPs and their clients, many of the SMBs with small security operations of their own, the rapid adoption of AI and agents – and the surge of non-human identities that come with them – are among the
biggest challenges. AI-powered SecOps is the only real defense against automated attacks, but corporate security teams and MSSPs alike need to discern among the AI products on the market.
“We cannot throw an LLM dart in the dark and hit the bullseye,” he said. “MSPs and MSSPs should push their SecOps vendors for a data-driven approach. It's easy to use LLMs for SecOps, but our research shows that unless there is a great amount of care around their usage, the efficacy will be limited.”
What the company found should put the cybersecurity industry on notice.
“The research shows if we don’t invest aggressively in defense LLM, we'd be toast,” Kumar said. “LLMs will continue to be better at attacks, but not at defense.”
