Methodology

Overview

Each lab is built from our own security research, led by BB Arena's creator Marius du Preez, and recreates the attack path behind a real-world vulnerability in a fully synthetic, isolated environment. Each model is dropped in black-box as an autonomous agent: it gets a target, a scope and a shell, and has to find the weakness, exploit it and capture proof.

• Proof, not claims. The lab's own verifier watches every attempt out of the agent's sight, and every lab instance carries unique proof values that only a real exploit can obtain. If the proof isn't in the agent's own output, it didn't happen.
• Same conditions for every model. One harness, one tool, the same labs, the same limits. Results compare models, not agent products.
• Partial progress counts too. Every lab is a four-rung ladder, so an agent that gets halfway still shows how far it got.

BB Arena v1 and v2

BB Arena v2 is a new benchmark. None of its labs were in BB Arena v1, and scores are much lower than on v1. The harness, the time limit, the two-attempt protocol and the proof rule are the same as v1. What changed is the labs: a new, harder set of 100, built and checked to a stricter standard, and every lab a strict four-step ladder in which each rung needs the one before it.

• Results on the two versions are never mixed or ranked together. A model's v1 and v2 scores aren't comparable, because the labs and their difficulty differ.
• BB Arena v1 is complete and no longer updated. Its leaderboard, model pages and methodology stay online (see the v1 leaderboard), and the v1 | v2 switch at the top of the page takes you between the two versions of the same page.
• Because full solves are rarer on v2, results also show how many rungs each model proved in every lab, overall and per vulnerability class.

Why black-box

Most cyber benchmarks for AI give the model the application's source code to read (white-box), or reuse public capture-the-flag challenges. That measures something useful, but it isn't what an outside attacker or a bug-bounty hunter faces: they get a running target and nothing else. BB Arena tests exactly that situation.

Public benchmarks have a second problem: their tasks, and often worked solutions, are on the internet. Once they end up in training data, scores can rise because a model has seen the answers rather than because it got better at security. Our labs, prompts and verifiers are never published, and every lab's identities and proof values change on every run, so there are no answers for a model to have learned.

The labs

BB Arena v2 is 100 new labs built from our own security research across many different applications, written in 7 programming languages. None of them were in BB Arena v1, and none shares an application with v1's labs. Each lab captures only the path to a vulnerability: how an attacker gets there and what it gives them. The application around it is fully synthetic and written from scratch, with no real organisation names, domains, credentials, requests, customer data or code.

The labs are grouped into 10 vulnerability classes, exactly 10 labs in each, so every area carries the same weight. The 100 are frozen as a fixed set: every published BB Arena v2 result uses exactly the same labs.

A wider attack surface

• Many labs are small systems rather than a single app, with internal services and background workers the agent can't reach directly.
• Browser-side labs include a simulated user whose browser opens what the agent plants.
• AI-agent labs are products with AI-assistant features, such as tool servers, assistant memory and retrieval, where the weakness is in how the assistant's tools and data are exposed.
• Some labs let the agent gain control of a component (run code, read files, make the server send requests) and continue from there.
• Every flaw sits among correctly secured features of the same shape, and nothing in the app points at the broken one: the agent has to find which one is wrong.

Every lab was screened for difficulty before it was included: labs a strong current model solved easily were made harder or left out. That's why scores are much lower than on BB Arena v1 and full solves are rare.

Rankings always use the whole benchmark. Each model's page also shows how it did in every vulnerability class, as totals: individual labs are never published.

Isolation and integrity

• Certified solvable. Before a lab is used, a reference exploit has to earn all four rungs against it. An unsolved lab is the agent's miss, not a broken lab.
• Fresh and unique every time. Every attempt starts from a clean environment. Identities, secrets and proof values are regenerated on every run, so proofs can't be reused or memorised.
• Hidden verification. The lab records security-relevant events out of the agent's sight. The agent can't see or influence what the verifier logs.
• Locked-down network. Labs run on a dedicated container runtime. The agent can reach only its target and a per-run model proxy, and model API keys never enter the agent's environment.
• Certified build. Each lab's certificate is tied to its exact build, and any change to a lab voids its certificate until it is checked again. Results are recorded against the certified build.
• Reference exploit and patched twin. A reference exploit must earn all four rungs on the lab, and must fail on a patched twin of the same lab, across several fresh instances.
• Every rung reached by an agent. Beyond the reference exploit, every rung of every lab has been reached in internal testing by an AI agent attempt confirmed by the lab's verifier on the certified build. An unsolved rung is the model's miss, not an unreachable step.
• Independent review. Every lab's design and its finished build are reviewed by a reviewer independent of whoever built it, checking for shortcuts, giveaways and broken scoring.
• Scoring outside the agent's reach. Where a lab lets the agent take over a component, proof values, keys and the verifier's record live outside every component the agent can control, and nothing a compromised component sends can award a rung. Each such lab is tested for this from inside the compromised component.

What the agent gets

The harness

The harness is the program that drives the model: it sends the instructions, runs the commands the model asks for and feeds the results back. Every model on BB Arena runs in the same harness, built by us, so the only thing that changes between results is the model.

Why our own harness, not each vendor's agent tool

Most AI labs ship their own agent product, such as Claude Code, Codex or Grok Build. Each one is built and tuned for its own vendor's models, with its own system prompt, tools, context handling and safety settings. Comparing models inside their own products would mix up two things: how good the model is, and how good the product around it is. It would also favour whichever vendor's tooling happens to suit this kind of work, and make results impossible to compare across vendors. A single neutral harness removes that bias.

What every model gets

• Exactly one tool: a Bash shell inside its isolated lab environment. No vendor-specific tools or plug-ins.
• The same instructions and scope, sent through the same provider-neutral interface.
• The same 25-minute time limit per attempt, and no other limit: no turn cap and no token or cost budget.
• The model maker's own API, never a reseller or router. Third-party providers can serve a quantised or otherwise altered copy of a model, or run it more slowly.
• Its reasoning carried forward between turns where the provider returns it. We never add reasoning a provider doesn't return.
• The same context handling: at 85% of the model's context window, older work is compacted and the most recent steps are kept intact. Context limits come from the provider, not a fixed list.

Common questions

Wouldn't a model score higher in its own vendor's tool?

It might, and that's exactly why we don't use those tools for results: the score would then reflect the product, not the model. We do run native tools internally as a diagnostic, but those runs are never published and never averaged with these results.

Is the harness tuned for any particular model?

No. It isn't adjusted per model or per vendor. Every model gets the same tool, instructions and limits; the only per-model settings are the effort level being tested and the context window, taken from the provider.

Why only a shell?

A shell is the common ground every capable model can use, and it's how a real tester works. Whatever command-line tools the lab environment provides are reachable through it, in exactly the same way for every model, rather than through vendor-specific tool integrations that some models are trained for and others aren't.

Why only the model maker's own API?

Resellers and routers can serve a model differently from its maker: quantised to cut cost, on slower hardware, or with lower limits. Within a fixed time limit that changes the result, so every model is tested exactly as its maker serves it.

A configuration is one model at one effort level. The same model at a different effort level is listed separately. Vendors' effort labels ("high", "xhigh", "max") aren't comparable units of compute between vendors.

Rungs and scoring

Every lab is a strict four-rung ladder: 400 rungs across the benchmark. Rung 2 can only be earned after rung 1, rung 3 after rung 2 and rung 4 after rung 3, and each rung uses something the agent obtained at the rung before: a value, an access or a foothold. Each rung needs its own reasoning step, not a repeat of the one before.

1. 1First break

   Exploit the weakness once and cross a security control: another user's data, a privileged action, an internal resource.

2. 2Use what you got

   Take what rung 1 gave you and get past the next control.

3. 3Go deeper

   Turn that into access to consequential data, state or a component of the system.

4. 4Complete the chain

   Finish the attack end to end.

Rung 1 is already a real exploit step. Ordinary use of the product never earns a rung: every lab is checked so that an ordinary user going about normal tasks proves nothing.

A solve means all four rungs were proven. Configurations are ranked by solves, and rungs proven break ties. Rungs proven also show partial progress on labs an agent didn't fully solve.

A worked example

Our real labs stay private, so here is a made-up one to show how scoring plays out. In BB Arena v2 it would sit in the access control class.

• Stops after step 2: 2 rungs proven, no solve.
• Claims it took over the admin account but has no step-4 proof value: step 4 earns nothing, so 3 rungs proven.
• Proves all four: 1 solve, 4 rungs proven.

Run protocol

1. Step 1First attempt

   All 100 labs, one attempt each.

2. Step 2Retry

   Only labs not fully solved on the first attempt, with a new sampling seed and a fresh conversation.

3. ResultFinal score

   Everything proven across both attempts. Labs solved the first time aren't run again.

• The retry starts from scratch, with no memory of the first attempt.
• Retries run as soon as a lab's first attempt finishes, rather than waiting for the whole first round.
• An attempt that fails for infrastructure reasons is re-run identically, up to three times, and never counts against the model.
• Effort level is set before the run and doesn't change during it.

What gets published

• Only complete runs, where every one of the 100 labs reached a final result. Partial runs, and runs cut short by provider rate or quota limits, aren't published. A run where safety refusals stopped attempts is complete, so it is published.
• Each configuration is run once, and that complete run is what's published.
• Attempts stopped by a provider's safety refusals count as unsolved (see Safety refusals).
• A BB Arena v2 result needs all 100 of its labs resolved. BB Arena v1 and BB Arena v2 results are never aggregated, averaged or ranked together, and each version has one published run per model and effort level.

Safety refusals

Some AI providers run cyber safeguards that stop requests they classify as offensive security work. BB Arena is enrolled in the cyber programs Anthropic and OpenAI offer to verified security researchers. A model runs with that access only where the provider's programme covers it at our tier; models the programme doesn't cover, and other providers' models, run with standard API access. On BB Arena v2 charts, configurations tested with this verified cyber access have a checkered fill, and each one's model page names the programme and tier. It's the same model as the public API, with the provider's cyber safeguards relaxed for verified security work. Refusals can still happen. When they do, they're part of the result, because that's what a security team using the model would get.

• An attempt the provider refuses scores zero.

• A lab whose first attempt was stopped still gets its retry, like any other missed lab.
• Scores are shown exactly as earned, and the leaderboard has a Refusals column: the number of labs where at least one attempt was stopped.
• When a model has refusals in more than 10 of the 100 labs, it isn't ranked: its score says more about the provider's safeguards than about the model, and its cost, token and time figures only cover part of the benchmark. It moves to the refusals board at the bottom of the front page, with the score it earned, how many labs had refusals and what the provider did. A note under the score chart and the leaderboard points to it.
• Ranked models with a few refusals are marked with a warning sign and listed on the refusals board too. Every model's page shows how many labs had refusals, and in which vulnerability classes.
• A run is published even when its provider's safeguards stop every attempt, on the refusals board with the score it earned. A low score from refusals reflects the provider's policy, not a judgement on the model's capability.

Cost and tokens

Cost is the total API spend for the whole benchmark, across both attempts: input, cached input, cache writes and output. Each lab attempt is priced from published per-token rates at the moment it finishes, and that price is stored with the result, so later price changes never re-price a run.

• A model without a published list price is left out of cost comparisons and shown as Unpriced. If it's temporarily free, for example during a launch promotion, we say so rather than treating it as $0.
• Infrastructure re-runs aren't included.
• When a model has safety refusals in more than 10 of the 100 labs, it's left out of the cost charts (see Safety refusals).
• Cost doesn't include infrastructure, storage, harness compute, engineering time, human review, taxes or subscriptions.

Speed and reliability

Speed figures describe one agent working on one lab, so they reflect the model and the provider endpoint used, not how many agents we ran in parallel.

• Time per lab: average wall-clock time of one lab attempt, across both attempts.
• Output speed: output tokens per second for a single agent.
• Response time: median across every model request in the run.
• API success rate: share of model requests that completed without a provider error.

Metric glossary

Solves

Higher is better · labs solved, of 100

Labs where the agent proved all four rungs. A rung counts as proven only when the lab's own verifier saw the action and the agent's own trace contains the exact proof for that lab instance. Final result after the first attempt and the retry.

Rungs proven

Higher is better · rungs proven, of 400

Each lab is a strict four-rung ladder, from the first break to completing the chain: 400 rungs in total. Each rung needs the one before it, so rungs proven show how far up each lab's ladder the agent got, including on labs it didn't fully solve, and break ties between equal solve counts.

First-attempt solves

Higher is better · solved on the first attempt

Labs solved on the first attempt, before any retry.

Safety refusals

Lower is better · labs with refusals, of 100

Labs where the provider's cyber safeguards stopped at least one attempt, by refusing or by answering with a different model. Stopped attempts score zero: work done by a fallback model never counts.

Cost to run

Lower is cheaper, not necessarily better · USD, full benchmark

Total API spend for the whole benchmark (both attempts), priced at the provider's list prices on the test date, including cache pricing. Excludes infrastructure. On its own a low cost can just mean few solves, so read it with cost per solve or the value chart.

Cost per solve

Lower is better · USD per solved lab

Cost to run divided by solves: what one fully proven finding cost.

Tokens used

Fewer is cheaper, not necessarily better · total tokens, full benchmark

All tokens processed across both attempts: input, cached input and output. Using few tokens can also mean the agent gave up early, so read it alongside solves.

Cache rate

Higher is better · share of tokens read from cache

Share of all tokens served from the provider's prompt cache. Higher usually means cheaper long agent sessions.

Time per lab

Lower is faster, not necessarily better · avg minutes per lab attempt

Average wall-clock time one agent spent on one lab attempt. Each attempt is capped at 25 minutes. A quick time can also mean the agent gave up early, so read it alongside solves.

Output speed

Higher is better · output tokens per second, one agent

How fast a single agent receives output tokens from the provider while generating. Reflects the model and the provider endpoint used.

Response time

Lower is better · median, full request

Median time for a full model response, from request to last token.

API success rate

Higher is better · requests that succeeded

Share of API requests that completed without a provider error. A reliability signal for running agents unattended.

API error rate

Lower is better · requests that failed

Share of API requests that failed with a provider error (the inverse of API success rate).

Limitations

• Labs are realistic but synthetic: no web application firewalls, scope ambiguity, duplicate reports or triage.
• Every benchmark lab is vulnerable. The secure twins that come with licensed labs aren't part of the benchmark yet, so it doesn't yet measure whether an agent invents findings that aren't there.
• BB Arena v2 is new: 15 configurations are published so far, and more are added as models are tested.
• Each vulnerability class has 10 labs, so one lab is 10% of a class and per-class numbers swing on single results.
• It's built to be hard: full solves are rare, so rungs proven carry most of the signal.
• Each result is a single run, and two attempts per lab are not a statistical pass@k estimate.
• Speed figures reflect the provider endpoint we used at the time and can differ elsewhere.
• Prices are a snapshot from the test date; the same run could cost more or less today.

Citing BB Arena

If you use BB Arena results in research, reporting or a product page, please cite the benchmark and the date you accessed it.

@misc{bbarena2026,
  author       = {du Preez, Marius},
  title        = {{BB Arena}: A Black-Box Cyber Benchmark for {AI} Models},
  year         = {2026},
  publisher    = {MDP SEC},
  howpublished = {\url{https://blackboxarena.ai/}},
  note         = {BB Arena v2}
}

Public and private