BB Arena
v1v2New

bbarena v2.0 · updated 14 Nov 2026

Which AI models can find and prove real vulnerabilities?

BB Arena runs each AI model as an autonomous agent, black-box, against 100 web and API labs built from our own security research. No source code, no hints, and private labs that models can't have trained on. Each lab is a strict four-rung ladder, from the first break to completing the chain, and only counts as solved when the agent proves all four.

Claude Opus 5 (High) leads with 22 of 100 labs solved, ahead of Grok 4.6 (High) on 21.

Top score22/100Claude Opus 5 (High)Best value$1.59 per solveDeepSeek V4.1 Flash (Max) · 21 solvesBest first attempt17 solved first timeDeepSeek V4.1 Flash (Max)The benchmark100 labs · 400 rungs15 configurations tested
All models
    • 22
    • 0
    • 21
    • 12
    • 5
    • 8
    • 6
    • 8
    • 21
    • 18
Colour

Score

How many of the 100 black-box labs each model fully exploited and proved.

labs solved, of 100·Higher is better
  • Verified cyber access
  • First attempt
  • Added on retry
What this measures

Labs where the agent proved all four rungs. A rung counts as proven only when the lab's own verifier saw the action and the agent's own trace contains the exact proof for that lab instance. Final result after the first attempt and the retry.

5 configurations aren't ranked: safety refusals stopped them in more than 10 of 100 labs. See the refusals board

Leaderboard

Every ranked configuration: one model at one effort level, all in the same neutral harness.

Model
1Claude Opus 5 HighAnthropic · Proprietary221780$923$41.97705.3M24.3 min71.2 tok/s99.27%
2Grok 4.6 HighxAI · Proprietary211750$399$19.01481.0M23.6 min74.8 tok/s98.52%
3DeepSeek V4.1 Flash MaxDeepSeek · Open weights211680$33.42$1.594.66B24.7 min145.5 tok/s99.74%
4Grok 4.7 HighxAI · Proprietary181520$748$41.551.23B16.6 min79.1 tok/s99.94%
5GPT-5.6 Sol HighSafety refusalsOpenAI · Proprietary121291$194$16.19241.4M10.3 min45.3 tok/s99.97%
6MiMo V2.6 Pro HighXiaomi · Open weights81120$41.76$5.22740.4M24.4 min59.3 tok/s98.90%
7GLM-5.3 Flash MaxZhipu AI · Open weights81020$36.13$4.52911.4M25.1 min55.5 tok/s99.24%
8MiMo V2.6 Flash HighXiaomi · Open weights6860$8.22$1.37541.2M25.3 min52.9 tok/s98.57%
9GPT-6 Sol HighOpenAI · Proprietary5880$45.03$9.0189.4M5.9 min35.2 tok/s99.97%
10Claude Haiku 5.5 HighAnthropic · Proprietary0180$2.73n/a44.1M2.5 min194.2 tok/s99.98%

Ranked by solves (of 100); rungs proven break ties. Bar: first attempt, lighter segment = added on retry. Click a column to sort, a model for its full profile. Refusals: labs where the provider's cyber safeguards stopped an attempt. Stopped attempts score zero and we never count a fallback model's work; with many refusals, cost, tokens and time are left out of comparisons (why).

5 configurations aren't ranked: safety refusals stopped them in more than 10 of 100 labs. See the refusals board

How far each model got

Each bar is one model's 100 labs, split by how many of the four rungs it proved in each. Solves are the labs at 4 of 4; the rest shows how far it got where it didn't finish.

  • Rungs proven per lab
  • 0 of 4
  • 1 of 4
  • 2 of 4
  • 3 of 4
  • 4 of 4 (solved)
labs by rungs proven, of 100 · More dark blue is better

By vulnerability class

Labs solved per type of vulnerability, to compare models where it matters to you. Each model's page has the full detail.

ModelAccess control10 labsAuth & sessions10 labsInjection10 labsCode execution10 labsServer-side10 labsBrowser-side10 labsInfra & edge10 labsLogic & races10 labsSecrets & crypto10 labsAI agents10 labs
Claude Opus 5 High3/102/101/102/101/102/102/101/106/102/10
Grok 4.6 High5/103/101/103/101/100/102/103/103/100/10
DeepSeek V4.1 Flash Max6/103/103/100/101/101/101/102/103/101/10
Grok 4.7 High2/100/101/103/102/101/101/103/104/101/10
GPT-5.6 Sol High3/101/102/102/100/100/102/102/100/100/10
MiMo V2.6 Pro High3/101/100/101/101/101/101/100/100/100/10
GLM-5.3 Flash Max3/102/100/101/100/100/100/102/100/100/10
MiMo V2.6 Flash High2/101/100/100/100/100/101/101/101/100/10
GPT-6 Sol High2/100/100/100/100/102/100/101/100/100/10
Claude Haiku 5.5 High0/100/100/100/100/100/100/100/100/100/10
0% solved100%Best in class

Value

Solves against what it took to get them.

Up and to the left is better
  • Verified cyber access

Cost & tokens

What it cost to run the full benchmark, at list prices on the test date. Cost per solve accounts for results; cost to run on its own does not.

USD per solved lab·Lower is better
  • Verified cyber access
What this measures

Cost to run divided by solves: what one fully proven finding cost.

Speed & reliability

Measured per single agent, so the numbers reflect the model and its provider, not how many agents we ran in parallel.

avg minutes per lab attempt·Lower is faster, not necessarily better
  • Verified cyber access
What this measures

Average wall-clock time one agent spent on one lab attempt. Each attempt is capped at 25 minutes. A quick time can also mean the agent gave up early, so read it alongside solves.

Refusals board

Configurations whose provider's cyber safeguards stopped attempts in more than 10 of 100 labs aren't ranked: their score says more about the safeguards than about the model. Here's how each did on the labs it was never refused, against the best ranked model on those same labs.

Top bar: the model, in its vendor colour · Best ranked model on the same labsSolved, share of labs never refused · Higher is better
Claude Mythos 5.2 High 58 labs refused, despite CVP Defense Access19 of 4213 Grok 4.6
Claude Opus 5.5 High 32 labs refused, despite CVP Defense Access7 of 6817 Grok 4.6
Claude Sonnet 5.5 High 20 labs refused, despite CVP Defense Access1 of 8018 Opus 5
GPT-6 Astra High 76 labs refused0 of 248 Grok 4.6
Claude Fable 5.1 High 100 labs refusedRefused in every lab: nothing to compare

Ranked, with refusals in 10 labs or fewer: GPT-5.6 Sol (High), 1 lab, ranked #5

Stopped attempts score zero: we never count a fallback model's work. The labs a model was never refused aren't a random sample of the benchmark, so this compares models on the same labs; it doesn't predict a full score. How refusals are scored

Why BB Arena is different

Most cyber benchmarks hand the model source code or reuse public challenges. Real attackers get neither, and public tasks can leak into training data.

What the model gets
Source code to read (white-box), or puzzle-style capture-the-flag challenges.
A target, a scope and a shell. Black-box, like a real external security test.
The agent harness
Each model in its vendor's own agent tool, or a harness tuned for one model family.
One neutral harness for every model: the same tool, instructions, context handling and limits.
Where the tasks come from
Public tasks, often with solutions and write-ups published online.
100 fully synthetic labs built from our own security research into real-world vulnerabilities.
Training-data contamination
Public tasks and answers can end up in training data, so scores can rise without real skill improving.
Labs, prompts and verifiers are never published, and proof values change on every run, so there are no answers to learn.
What counts as success
Often the model's final answer or a single flag.
Verified proof at every step: the exact proof value and a server-recorded action, rung by rung.
Want your model on BB Arena?AI labs, providers and sponsors: we evaluate public and upcoming models under the same protocol as every other model.
Get your model tested →

How a lab is scored

Each lab recreates the attack path behind a real-world vulnerability in a fully synthetic, isolated environment. The agent gets a target, a scope and a shell. Nothing else.

  1. 1First breakExploit the weakness once and cross a security control: another user's data, a privileged action, an internal resource.
  2. 2Use what you gotTake what rung 1 gave you and get past the next control.
  3. 3Go deeperTurn that into access to consequential data, state or a component of the system.
  4. 4Complete the chainFinish the attack end to end. All four proven = one solve.

A rung counts as proven only when the lab's own verifier saw the action and the agent's own trace contains that lab instance's unique proof. What the agent claims earns nothing. Every configuration gets up to two attempts at each lab; the retry only runs on labs missed the first time, and starts fresh. Full methodology →