Cyber Model Arena
We evaluate AI agents with foundation models and standard harnesses on multiple real-world cybersecurity tasks.
Model Quality vs. Efficiency
Success rate against cost, time, or steps per solve over k independent attempts.
Leaderboard
Each percentage is pass@1 in that category.
# | Agent | Overall @ 1 | Code Vulns | API Sec | Web Sec | Cloud Sec | Efficiency / pass |
|---|---|---|---|---|---|---|---|
| 1 |  ADK React | 77.2% | 65.8% | 85.8% | 74.7% | 82.6% | 4.3 min$0.7715 steps |
| 2 | Antigravity | 77% | 64.9% | 87% | 69% | 87% | 8.4 min$0.9751 steps |
| 3 | ADK React | 74.9% | 57% | 85.2% | 90.8% | 66.7% | 5 min$1.6258 steps |
| 4 |  Claude Code | 71.2% | 55.2% | 84% | 81.6% | 63.8% | 12.2 min$2.6319 steps |
| 5 |  Codex | 71.2% | 55.1% | 78.4% | 86.2% | 65.2% | 6.6 min$0.6721 steps |
| 6 | Claude Code | 70.8% | 59.9% | 82.7% | 66.7% | 73.9% | 5.3 min$0.8416 steps |
| 7 | Claude Code | 70.7% | 56.6% | 80.2% | 79.3% | 66.7% | 8.1 min$1.2316 steps |
| 8 | Codex | 70.6% | 56.4% | 80.9% | 77% | 68.1% | 4.9 min$0.4418 steps |
| 9 | Claude Code | 70.2% | 56.6% | 81.5% | 70.1% | 72.5% | 6.8 min$6.4816 steps |
| 10 | Codex | 69.8% | 57.2% | 79.6% | 77% | 65.2% | 5.7 min$2.0622 steps |
| 11 | Claude Code | 69.3% | 50.8% | 78.4% | 88.5% | 59.4% | 17.9 min$1.2651 steps |
| 12 | Codex | 69.2% | 56.1% | 81.5% | 71.3% | 68.1% | 4.1 min$2.6818 steps |
| 13 | Claude Code | 68.5% | 50.1% | 84.6% | 78.2% | 60.9% | 5.7 min$1.5258 steps |
| 14 | Antigravity | 68% | 48.7% | 84% | 77% | 62.3% | 4.3 min$0.89104 steps |
| 15 | ADK React | 66.9% | 41.8% | 78.4% | 85.1% | 62.3% | 15.3 min$7.4612 steps |
| 16 | Antigravity | 65.6% | 45.6% | 80.2% | 71.3% | 65.2% | 7 min$0.4774 steps |
| 17 | Claude Code | 65.4% | 44% | 79% | 74.7% | 63.8% | 11.9 min$2.7715 steps |
| 18 | ADK React | 64.7% | 58.7% | 75.9% | 57.5% | 66.7% | 5.3 min$2.5612 steps |
| 19 | Claude Code | 64.5% | 58.5% | 77.8% | 55.2% | 66.7% | 3.2 min$0.789 steps |
| 20 | Codex | 64.5% | 46.8% | 72.8% | 80.5% | 58% | 6.3 min$0.0518 steps |
| 21 | Claude Code | 64.4% | 38.9% | 78.4% | 79.3% | 60.9% | 22.4 min$3.4847 steps |
| 22 | Claude Code | 64.2% | 33.8% | 82.7% | 79.3% | 60.9% | 27.2 min$8.2957 steps |
| 23 | Codex | 63% | 48.9% | 75.3% | 65.5% | 62.3% | 4.6 min$0.9220 steps |
| 24 | ADK React | 61% | 25.8% | 80.2% | 81.6% | 56.5% | 30.6 min$2.3139 steps |
| 25 | ADK React | 60.9% | 35.3% | 77.2% | 75.9% | 55.1% | 15.7 min$4.1612 steps |
| 26 | Claude Code | 60.5% | 37.7% | 73.5% | 81.6% | 49.3% | 22.5 min$8.9947 steps |
| 27 | Claude Code | 60.5% | 43.4% | 75.3% | 71.3% | 52.2% | 21.1 min$0.0930 steps |
| 28 |  Claude Code | 60.3% | 37.1% | 83.3% | 58.6% | 62.3% | 24.4 min$0.1747 steps |
| 29 | ADK React | 59.7% | 17% | 80.2% | 85.1% | 56.5% | 38.7 min$0.9537 steps |
| 30 | Antigravity | 59.3% | 38.9% | 77.2% | 63.2% | 58% | 7.4 min$1.3178 steps |
| 31 | Claude Code | 59.3% | 36.3% | 76.5% | 67.8% | 56.5% | 14.1 min$1.5816 steps |
| 32 | Claude Code | 59.1% | 26.1% | 82.1% | 74.7% | 53.6% | 27.4 min$6.4738 steps |
| 33 | Claude Code | 59.1% | 39.4% | 75.9% | 69% | 52.2% | 16 min$9.3240 steps |
| 34 | ADK React | 58.7% | 31.4% | 81.5% | 60.9% | 60.9% | 26.3 min$2.3719 steps |
| 35 |  Claude Code | 58% | 32.9% | 82.7% | 65.5% | 50.7% | 28 min$3.1333 steps |
| 36 | ADK React | 57.6% | 34.7% | 82.7% | 60.9% | 52.2% | 26.8 min$0.1544 steps |
| 37 | Claude Code | 57% | 46.6% | 77.8% | 52.9% | 50.7% | 6.6 min$0.2218 steps |
| 38 | Claude Code | 54.4% | 37.9% | 72.8% | 62.1% | 44.9% | 12.1 min$0.6720 steps |
| 39 | ADK React | 53.6% | 17.3% | 74.7% | 75.9% | 46.4% | 41.6 min$2.3546 steps |
| 40 | ADK React | 52.8% | 27.5% | 77.2% | 52.9% | 53.6% | 31.9 min$1.6149 steps |
| 41 | ADK React | 52.5% | 36.3% | 76.5% | 44.8% | 52.2% | 19.2 min$1.5317 steps |
| 42 |  ADK React | 52.2% | 27.5% | 72.2% | 59.8% | 49.3% | 21.7 min$1.0920 steps |
| 43 | Claude Code | 51% | 26.5% | 75.9% | 60.9% | 40.6% | 28.6 min$1.8921 steps |
| 44 | Claude Code | 50.5% | 32% | 68.5% | 50.6% | 50.7% | 17.7 min$2.3932 steps |
| 45 | Claude Code | 50.2% | 21% | 78.4% | 44.8% | 56.5% | 39.4 min$0.4131 steps |
| 46 | ADK React | 49.1% | 21.7% | 75.3% | 51.7% | 47.8% | 40.1 min$1.4028 steps |
| 47 | ADK React | 48.6% | 34.1% | 76.5% | 33.3% | 50.7% | 7.1 min$0.0516 steps |
| 48 | Claude Code | 48.2% | 24.1% | 77.2% | 49.4% | 42% | 33.9 min$0.3424 steps |
| 49 | ADK React | 47.4% | 16.4% | 80.9% | 41.4% | 50.7% | 40.1 min$0.2228 steps |
| 50 |  ADK React | 46.6% | 23.4% | 56.2% | 67.8% | 39.1% | 43 min$2.3916 steps |
| 51 | ADK React | 46.5% | 19% | 71% | 48.3% | 47.8% | 46 min$1.7559 steps |
| 52 | Claude Code | 45.5% | 9.7% | 71.6% | 64.4% | 36.2% | 51.2 min$2.8515 steps |
| 53 | Antigravity | 44.3% | 21.9% | 72.2% | 42.5% | 40.6% | 15.7 min$1.0781 steps |
| 54 | ADK React | 42% | 14.3% | 77.2% | 29.9% | 46.4% | 42.5 min$1.6926 steps |
| 55 | ADK React | 41.8% | 12.4% | 78.4% | 35.6% | 40.6% | 55.5 min$0.2020 steps |
| 56 |  ADK React | 39.7% | 15.2% | 77.8% | 31% | 34.8% | 55.5 min$0.1717 steps |
| 57 | Claude Code | 39% | 13.6% | 76.5% | 29.9% | 36.2% | 58.7 min$0.5314 steps |
The state of agentic offensive AI
Agents can already perform standard cyber tasks
Leading agents already complete most of our basic, real-world security challenges on their first attempt.
- Gemini 4Argon +ADK React77.2%
- Gemini 4Argon +Antigravity77%
- Gemini 3.8Flash Cyber +ADK React74.9%
- ClaudeOpus 5 +Claude Code71.2%
The same model performs differently across harnesses
The same model can perform substantially better or worse under Claude Code vs ADK React. We evaluate the pairing, not the model alone.
- Claude Sonnet 5 +ADK React42%
- Claude Sonnet 5 +Claude Code59.3%
Attacks are inexpensive and take minutes
A successful AI agent attack typically costs a few dollars and completes within minutes. Additional attempts (e.g., pass@3) increase the attack success rate.
Code vulns are still hard for agents to find and exploit
We cover many challenges of finding vulns in a large codebase and 1-day (known-CVE) exploitation. We find such tasks to be harder for agents to solve.
- API Security85.8%
- Cloud Security82.6%
- Websec.fr CTFs74.7%
- Code Vulns65.8%
Datasets
Code Vulns
221 challenges
Identify known vulnerability patterns in source code, and exploit 1-days (known-CVEs).
Identify the vulnerability root cause in Python, Java and Go, or exploit a live 1-day CVE.
API Security
54 challenges
Discover and exploit web vulnerabilities through live API.
Reach all exploitable paths in labs using no source code, and often no API spec.
Websec.fr CTFs
29 challenges
Analyze Web CTF challenges source code and write working exploits to capture flags.
Use the public websec.fr website tasks.
Cloud Security
23 challenges
Exploit misconfigurations across different cloud providers (taken from Wiz CTF Challenges).
Capture a flag within AWS / Azure / GCP / Kubernetes-style setups.
Harnesses
ADK React
A simple ReAct loop developed with ADK. It uses a single bash tool, has no specialized cyber tools, and uses a general system prompt.
Claude Code
Anthropic's standard coding CLI. It uses ordinary file, search and shell tools, has no specialized cyber tools, and uses a general system prompt.
Antigravity
Google's Antigravity CLI. It uses ordinary file, search and shell tools, has no specialized cyber tools, and uses a general system prompt. Published only with Google models.
Codex
OpenAI's Codex CLI. It uses ordinary file, search and shell tools, has no specialized cyber tools, and uses a general system prompt. Published only with OpenAI models.
Methodology
The agent receives the task instruction only, with no extra attack playbooks or exploit templates.
Each harness-model-challenge combination is run 3 times (pass@3: best result across runs is taken per challenge), with a retry limit of 3 per time for infra errors.
Each agent run is limited to 1000 seconds. When time expires, the agent is stopped and the verifier scores whatever the agent produced. A timeout is not retried.
Reasoning level is set to high across models.
Anti-cheat: agents run in isolated Docker containers with no CVE databases and no external resources.
All scoring is deterministic via API hooks, exact flags, exact root-cause strings, and asserting the vulnerable path is executed.
The overall score is the macro-average across all 4 categories.