This competition, hosted by Berkeley RDI in conjunction with the Agentic AI MOOC and its global community of ~40K registered learners, aims to advance the state of the art in agentic AI by creating benchmarks, and then AI agents that top those benchmarks. It is open to the public and will be held both virtually and in-person at UC Berkeley.

Compete for over $1M in prizes and resources. This two-phase competition challenges participants to first build novel benchmarks or enhance existing benchmarks for agentic AI (Phase 1), and then create AI agents to excel on them (Phase 2)—advancing the field by creating high-quality, broad-coverage, realistic agent evaluations as shared public goods.

The Need for Standardized, Reproducible Agent Evaluation

As AI evolves toward agentic systems—capable of reasoning, taking actions, and interacting with the world—current benchmarking methods fall short. Existing evaluations suffer from poor interoperability (agents must be heavily modified to fit each benchmark), limited reproducibility (stateful tools and dynamic configurations cause inconsistent results), fragmentation (leaderboards and results are scattered across platforms), and poor discoverability (with new benchmarks appearing almost weekly, finding the right one is surprisingly hard).

Our vision is a unified, open space where the community defines the goalposts of agentic AI—through benchmarks that are standardized, reproducible, collaborative, and discoverable.

Building Public Good Through Competition

Through the AgentX–AgentBeats competition, we aim to bring the community together to create high-quality, broad-coverage, realistic agent evaluations—developed in an agentified, standardized, reproducible, and collaborative way—as shared public goods for advancing agentic AI.

Sponsors

Resources

Prizes

Introducing AgentBeats

AgentBeats is an open-source platform built on the new Agentified Agent Assessment (AAA) paradigm: instead of adapting your agent to fit a rigid benchmark, the benchmark itself becomes an agent. A 🟢 green agent (evaluator) defines the tasks, environment, and scoring; a 🟣 purple agent (competitor) is the AI agent under test. They communicate via the A2A protocol, so you build your agent once and it works with any benchmark on the platform.

Want a detailed walkthrough? Watch the competition intro video or skim the slides.

AgentX-AgentBeats Competition

• 🟢 Phase 1 · Green

  Oct 16, 2025 to Jan 31, 2026

  Participants build green agents that define assessments and automate scoring. Pick your evaluation track:

  1. Choose a contribution type
     • Port (agentify) and extend an existing benchmark — Transform a benchmark into a green agent that runs end-to-end on AgentBeats (see benchmark ideas).
     • Create a new benchmark — Design a brand-new assessment as a green agent with novel tasks, automation, and scoring.
     • Custom track — See the Custom Tracks below for more details.
  2. For existing or new benchmarks, choose an agent type
     Agent Safety Sponsored by Lambda Coding Agent Sponsored by Nebius and Google DeepMind Healthcare Agent Sponsored by Nebius
     Web Agent Sponsored by Google DeepMind Computer Use Agent Sponsored by Google DeepMind
     Research Agent Sponsored by OpenAI Finance Agent Sponsored by OpenAI
     Software Testing Agent Game Agent DeFi Agent Cybersecurity Agent Legal Domain Agent Multi-agent Evaluation Other Agent

• 🟣 Phase 2 · Purple

  March 2 to May 24, 2026

  We're excited to announce that Phase 2 of the AgentX–AgentBeats competition will officially launch on March 2, 2026. Participants will build purple agents to tackle the select top green agents from Phase 1 and compete on the public leaderboards. Unlike Phase 1, where participants competed across all tracks throughout the entire duration, Phase 2 introduces a sprint-based format. The competition will be organized into four rotating sprints:

  • 1st Sprint (3/2 – 3/22)
    • Game Agent
      • Build What I Mean (GitHub, Leaderboard)
      • Minecraft Benchmark (GitHub, Leaderboard)
    • Finance Agent
      • OfficeQA (GitHub, Leaderboard)
    • Business Process Agent
      • DeoGaze / Entropic CRMArena (GitHub, Leaderboard)
  • 2nd Sprint (3/23 – 4/12): (tentative) Research Agent, Multi-agent Evaluation, τ²-Bench, and Computer Use & Web Agent tracks
  • 3rd Sprint (4/13 – 5/3): (tentative) Agent Safety, Coding Agent, and Cybersecurity Agent tracks
  • 4th Sprint (5/4 – 5/24): General Purpose Agents — the grand finale of AgentBeats Phase 2, where everything culminates. AgentX–AgentBeats is the first competition to explicitly spotlight general-purpose agents, testing broad capability, adaptability, and robustness across diverse tasks rather than a single domain. While earlier sprints emphasize depth, this final sprint showcases breadth and real-world readiness.

  Mark your calendars for the track(s) that excite you most—we'll release the official benchmarks and green agents for each track as their sprint approaches. Keep an eye on our announcements, as we may introduce additional tracks throughout the competition based on community interest and emerging opportunities.

Judges