The Best AI Agent Completes 51% of Real Work. Hand It a Human Plan and It Jumps 35 Points.

The benchmark that stopped grading demos

Most of the AI agent headlines you have read this year cite benchmarks built to be gamed, in a narrow, forgivable sense: single-turn coding puzzles, synthetic Q&A, tasks with one obviously correct answer. EnterpriseOps-Gymwas built by ServiceNow Research in collaboration with Mila and the Université de Montréal to be the opposite of that. It is a containerized enterprise sandbox: 164 database tables, 512 functional tools, live MCP servers standing in for the systems an actual operations team touches every day, across eight mission-critical verticals including HR, IT Service Management, Finance, Sales, and cross-domain hybrid work.

Agents are graded on 1,150 expert-curated tasks in oracle mode, where actions are irreversible inside the simulated environment and success depends on following policy, not just producing a plausible-looking answer. On July 8, 2026, Artificial Analysis published its independent run of the benchmark, EnterpriseOps-Gym-AA, scoring every major frontier and open-weight model against the same task set.

Fifty-one percent is not a typo

Here is the full leaderboard, best configuration per model:

Source: Artificial Analysis, “Announcing EnterpriseOps-Gym-AA”, July 8, 2026, and ServiceNow Research's EnterpriseOps-Gym project page.

The spread from first to sixth place is eleven points. That is a strikingly narrow band for a field that spans multiple labs, multiple architectures, and, as the next section shows, a 90x difference in what it costs to run a task. Every frontier lab currently ships a model that clears roughly the same real-world operations ceiling, and that ceiling sits at barely a coin flip.

The 79-51 gap: why almost right is still wrong

The single most important number in this benchmark is not on the leaderboard. It is the gap between two scores for the exact same model on the exact same run. EnterpriseOps-Gym grades roughly 12,000 individual assertions across its task set, things like “was the correct approval obtained before the record was modified” or “was the ticket routed to the right queue.” Claude Fable 5, the leaderboard winner, passes 79% of those individual checks.

But EnterpriseOps-Gym scores whole-task completion, not average check accuracy, and it uses all-or-nothing grading: a single missed constraint fails the entire task, even if every other step was executed perfectly. That is what turns a 79% check-pass rate into a 51% task-completion rate. The model is not failing at comprehension. It is failing at the last mile of compliance, the one skipped approval or one wrong field that a real business process treats as a hard stop, not a rounding error.

Where agents actually break: constraint density, not intelligence

EnterpriseOps-Gym breaks results down by business domain, and the pattern is not noise. It tracks almost exactly with how many policy constraints a domain's tasks carry.

A 32-point gap separates the easiest domain from the hardest, using the same set of models on the same day. This is the clearest evidence yet that “can this model handle enterprise work” is the wrong question. The right question is “how many hard constraints does this specific workflow carry,” because that number, not the model's benchmark score on MMLU or SWE-bench, is what predicts whether an agent will actually finish the job.

The most important finding in the whole paper

ServiceNow's researchers ran one additional experiment that the headline leaderboard numbers hide: they handed agents a plan a human expert had already written for a task, then let the agent execute that plan instead of forming its own. Nothing else changed. Same model, same tools, same underlying task, same grading.

Source: ServiceNow Research / Mila, “EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings,” arXiv 2603.13594.

A 14-to-35-point jump from swapping who does the planning is larger than the entire gap between the best and worst frontier models on the standard leaderboard. That reframes the whole benchmark. The industry narrative around agent failures usually centers on tool-calling errors, hallucinated function arguments, or context loss. EnterpriseOps-Gym shows those are secondary. Strategic planning, deciding what sequence of steps actually satisfies a multi-constraint objective, is the binding constraint, and it is measurable and addressable independent of which frontier model you use.

Here is the shape of the difference, illustrated as an agent harness would actually implement it:

# Standard run: the agent plans AND executes
result = agent.run(
    objective="Onboard new hire req #4417: provision accounts, assign equipment, "
              "route background-check approval, notify manager",
    mode="self-planned",
)
# ✓ OK  provisioned email + SSO account
# ✓ OK  equipment ticket opened
# ⚠ FIX approval routed to wrong queue (skipped HR policy check)
# → whole_task_completed: False   (one constraint miss = full task failure)

# Plan-first run: a human (or a dedicated planning pass) supplies the plan,
# the agent only executes it
plan = human_plan_for("onboard_new_hire", req_id=4417)
result = agent.run(objective=plan, mode="execute-only")
# ✓ OK  provisioned email + SSO account
# ✓ OK  equipment ticket opened
# ✓ OK  approval routed per HR policy, step order preserved
# → whole_task_completed: True
#
# Same model. Same tools. Same task. +14 to +35 points across EnterpriseOps-Gym's
# task set, because the planning step, not the execution step, was the failure point.

The cost paradox: paying 8x more for one point

Cost per task varies by roughly 90x across the field, and it does not track cleanly with score:

GLM-5.2, an open-weights model, scores within a single point of Claude Opus 4.8 for roughly an eighth of the price. Claude Fable 5 does earn a real, distinct lead at the top of the leaderboard, but the middle of the field shows almost no relationship between spend and outcome. For anyone deciding which model powers a given step of an agent pipeline, that is the more useful number than the raw leaderboard rank: cost-per-point-of- reliability, not cost-per-token.

Turn efficiency tells a related story. Claude Fable 5 completes tasks in an average of 6.5 turns; Gemini 3.5 Flash takes 13.5; Grok 4.5 takes 17.4, nearly three times as many moves for a lower success rate. The model at the top of the leaderboard is not just more accurate, it is measurably more decisive, which is itself a proxy for the same planning capability the human-plan experiment isolated directly.

What this means if you are actually building agents

The practical read is not “agents cannot do real work.” A 51% autonomous completion rate on a 1,150-task, 512-tool, all-or-nothing-graded enterprise benchmark is a genuine capability that did not exist two years ago. The read is about where to spend your engineering effort, and EnterpriseOps-Gym gives a specific, falsifiable answer: invest in the planning layer before you invest in a bigger model.

Concretely, that means separating “decide what to do” from “do it” as distinct stages in your agent pipeline, the same split ServiceNow's researchers tested directly:

# Two ways to structure the same agent job

# 1. Single-pass agent (plans and executes in one loop)
#    - simplest to build
#    - inherits the full 79%-checks / 51%-tasks gap from EnterpriseOps-Gym
agent.run(objective, mode="self-planned")

# 2. Plan-then-execute pipeline
#    - a dedicated planning pass produces an explicit, checkable plan
#      (by a reasoning-heavy model, a human reviewer, or both)
#    - a second, execution-only pass just follows it, with guardrails
#      checked at each step rather than only at the end
plan = plan_agent.run(objective, mode="plan-only", verify=is_checkable_plan)
result = exec_agent.run(plan, mode="execute-only", guardrails=domain_policy(objective))
# This is the structural change EnterpriseOps-Gym's +14 to +35 point result
# says is worth more than swapping which frontier model sits behind the API call.

It also means routing matters more than any single model choice. Constraint-dense domains like HR and ITSM are exactly where you want your most reliable planner in the loop, human-reviewed or otherwise, and low-constraint domains like email and file management are exactly where a cheaper, faster model with a tight execution harness gets you nearly the same result GLM-5.2 gets against Claude Opus 4.8 in the table above, for a fraction of the cost.

That is the kind of decision an always-on BrainClawagent is built to make continuously rather than once at design time. It runs on its own isolated VM with shell access, scheduling, and persistent memory, so a plan-then-execute pipeline can sit behind a cron job, watch its own guardrail-violation rate per domain, and flag when a workflow's constraint density has crept up enough to warrant a stronger planner. And when it is time to route the planning pass to a frontier reasoning model and the execution pass to something cheaper and faster, MegaBrain gives one API across 500+ models with transparent, at-cost pricing, so that split is a routing rule, not a rewrite.

Sign up at getmegabrain.com to route your agent fleet at cost, or spin up a BrainClaw agent to keep watching the planning-versus-execution gap in your own workflows long after EnterpriseOps-Gym's leaderboard numbers move again.