Frontier AI can ace a coding quiz, but can it actually finish an investment banker's deliverable? A benchmark released in January 2026 finally put a number on that, and the number is small.
How Mercor's APEX grades a professional deliverable
APEX-Agents is a benchmark that tests whether AI agents can autonomously complete long-horizon professional work to a client-ready standard, not just answer isolated prompts. Released by training-data company Mercor on January 21-22, 2026, it draws its tasks from senior practitioners at Goldman Sachs, McKinsey, and Cravath, covering investment banking, consulting, and corporate law . The point is to measure integrated knowledge work, not trivia.
Each task carries 1-10 expert-written pass/fail criteria, judged against a client-ready bar by domain experts who average 7+ years of experience . That rubric design matters: a deliverable either meets a professional's acceptance criteria or it does not, with no partial credit for sounding plausible.
The environment is what separates APEX from quiz-style evaluations. Every run happens inside a simulated Google Workspace backed by a Box file system, so an agent must retrieve files, reason across them, and synthesize a result before delivering . The full workflow is in scope:
- handling ambiguous, underspecified instructions;
- navigating across many files to find the right inputs;
- synthesizing across multiple documents and domains;
- producing a final, client-facing deliverable.
Mercor shipped it open source under CC-BY on Hugging Face and GitHub, scored with its open evaluation infrastructure, Archipelago; the authors are Brendan Foody, Bertie Vidgen, and Osvald Nitski . As TechCrunch framed the launch, the design deliberately probes whether agents are ready for real workplace tasks rather than benchmark set-pieces . Scores across models landed in a narrow band.
Where every AI placed on APEX: 24% per try, 40% after eight
No frontier model clears a quarter of APEX-Agents tasks on its first try. Gemini 3 Flash (Thinking=High) led at roughly 24%, GPT-5.2 followed at about 23%, while Claude Opus 4.5, Gemini 3 Pro, and GPT-5 clustered near 18% . Every score is a one-shot pass rate against a client-ready standard set by Goldman Sachs, McKinsey, and Cravath professionals .
| Model | One-shot pass rate | Domains evaluated |
|---|---|---|
| Gemini 3 Flash (Thinking=High) | ~24% | Investment banking · consulting · corporate law |
| GPT-5.2 | ~23% | Investment banking · consulting · corporate law |
| Claude Opus 4.5 | ~18% | Investment banking · consulting · corporate law |
| Gemini 3 Pro | ~18% | Investment banking · consulting · corporate law |
| GPT-5 | ~18% | Investment banking · consulting · corporate law |
The per-model scores are aggregate across all three verticals, not split by domain. APEX builds long-horizon tasks in investment banking, consulting, and corporate law, and reports a single pass rate per model .
Retries do not rescue the result. Granted eight attempts, the best agent reaches only about 40% of tasks . That leaves roughly 60% of these Goldman, McKinsey, and Cravath deliverables below an acceptable bar even when the model is allowed to try again and again.
"Are AI agents ready for the workplace? A new benchmark raises doubts." TechCrunch's framing of the APEX-Agents launch (source: TechCrunch, 2026-01).
The trajectory is steep. Mercor reports APEX scores rose from a 5-10% range a year earlier to 20%+ now . The direction is useful, but the absolute numbers will date fast.
For builders, that argues against pinning roadmaps to a single leaderboard snapshot. The ranking gap between the top model and the ~18% cluster is narrow enough to flip between releases. Plan for quarterly recalibration: re-run your own task suite each quarter rather than trusting a figure from two model generations ago.
What fails: ambiguity, navigation, and domain coordination
The breakage is not in reasoning. It is in workflow orchestration. Mercor's diagnosis of APEX-Agents is blunt: frontier models "can't manage ambiguity, find the right file, or hold context across the entire workflow" . Models that score well on isolated capability still collapse when a single task spans many documents and tools, because finishing a client-ready deliverable is an integration problem, not a quiz.
Agents "can't manage ambiguity, find the right file, or hold context across the entire workflow," and struggle most when tracking information across multiple domains and platforms simultaneously. Mercor, APEX-Agents (source: Mercor, 2026-01).
The hardest cases involve coordinating across several domains and platforms at once . The gap is agentic coordination, not raw intelligence, which is why peak benchmark numbers mislead anyone planning to hand off whole jobs.
The Remote Labor Index puts numbers on the failure shapes. Across attempts on 240 commissioned freelance projects, the breakdown was :
| Failure mode | Share of attempts |
|---|---|
| Below professional quality | 45.6% |
| Incomplete or malformed deliverable | 35.7% |
| File-integrity problems | 17.6% |
| Internally inconsistent across files | 14.8% |
The categories overlap, so they sum past 100%, but the pattern is clear . Most failures are not subtle quality misses; they are deliverables that arrive broken, incomplete, or self-contradictory across files. That maps directly onto APEX's "find the right file, hold context" deficit. Multi-file, long-horizon work is where agents lose the thread.
RLI is not uniform floor, though. The paper flags verticals where agents already produce acceptable output :
- Audio and image processing: bounded, single-artifact tasks.
- Writing: short-form text with a clear brief.
- Data retrieval and web scraping: well-specified extraction.
- Interactive data visualization: a defined input, a defined chart.
The common thread among the wins: a narrow input, a single output format, and little cross-document state to maintain. These verticals are addressable today; the failures cluster wherever a task demands tracking heterogeneous files, reconciling them, and holding intent across a long workflow, which is precisely the orchestration layer to build against.
Remote Labor Index: 240 freelance projects, 2.5% autonomous delivery
The Remote Labor Index (RLI) tests the same orchestration layer from the demand side: it asks whether an agent can deliver a whole commissioned project, not pass a scripted task. Built by researchers affiliated with the Center for AI Safety and Scale AI and posted in late October 2025, RLI scores agents against 240 genuine paid freelance projects worth $143,991 in total, representing over 6,000 hours of human professional effort . Individual projects ran past $10,000 and over 100 hours each .
The metric is automation rate: the share of projects where human evaluators judge the AI deliverable at least as acceptable as the human reference for a reasonable client. Frontier agents performed near the floor.
| Agent | Automation rate |
|---|---|
| Manus | 2.5% |
| Grok 4 | 2.1% |
| Claude Sonnet 4.5 | 2.1% |
| GPT-5 | 1.7% |
| ChatGPT agent | 1.3% |
| Gemini 2.5 Pro | 0.8% |
The best agent, Manus, autonomously completed 2.5% of projects; the rest landed below it . The result mirrors APEX's diagnosis: the bottleneck is finishing heterogeneous, long-horizon work, not raw capability on a single deliverable.
Two caveats matter for builders reading these numbers:
- Reproducibility is limited. 230 of the 240 projects are held private to limit contamination and protect privacy, leaving only 10 public examples to inspect or rerun .
- The live figure has moved, but is thin. CAIS's running dashboard now lists full automation around 4.17%, suggesting improvement since the paper . It does not disclose the model, run conditions, or confidence intervals, so treat it as an updated indicator rather than a replacement for the paper's table.
The conclusion is the same either way. Specified single-output tasks are tractable; autonomous delivery of a full freelance brief (ambiguous scope, mixed file types, client-acceptable quality) remains close to the floor. That gap, not peak benchmark scores, is what the next generation of agent tooling has to close.
GDPval at 47%: augmentation is not autonomous delivery
GDPval is the benchmark that explains why the autonomy numbers and the optimism can both be true. Released by OpenAI on September 25, 2025, it scored Claude Opus 4.1 as matching or beating the human deliverable on 47.6% of specified tasks across 44 occupations. That sounds like a different planet from RLI's 2.5%, until you read what each test actually asks.
GDPval measures output quality on a defined, single deliverable: produce this report, this analysis, this design, then have professionals rate it. Its 220-task open gold set was built by experts averaging over 14 years of experience, drawn from the top nine U.S. GDP-contributing industries. The claim it validates is augmentation: strong outputs for a clearly scoped ask.
APEX and RLI test the harder claim: autonomous completion of a whole project with heterogeneous files, ambiguous standards, and long horizons. That is automation, and the same model that hits 47.6% on a clean brief still finishes a fraction of full commissions.
OpenAI was explicit that the headline figure understates real-world cost.
"Models can be roughly 100x faster and 100x cheaper than the human expert baseline, but these estimates exclude oversight, iteration, and integration," OpenAI noted in its GDPval write-up (source: OpenAI).
Those excluded costs are exactly where APEX and RLI agents break. The multiplier on real deployment is unknown precisely because the integration layer is unmeasured here.
Two earlier studies corroborate the pattern:
- TheAgentCompany (Carnegie Mellon): in a simulated software firm of 175 workplace tasks, the best agent finished only 24.0% autonomously.
- QuestBench: across 256 research-domain questions, 13 evaluated deep-research systems averaged a 16.85% pass rate.
The split is consistent across every test. Scope the work tightly and frontier models are useful collaborators; hand them the full job and autonomous delivery collapses toward the floor.
What to target: reliability, grounding, and repeat-run consistency
The bottleneck is not peak single-turn quality. It is whether an agent finishes the full pipeline cleanly across repeated runs without dropping state, misidentifying files, or stalling on ambiguity. APEX-Agents traces failures to workflow competence, not raw reasoning: models "can't manage ambiguity, find the right file, or hold context across the entire workflow" . RLI tells the same story from its error breakdown: 35.7% of attempts shipped incomplete or malformed deliverables and 17.6% had file-integrity problems .
For builders, that points to three investment areas ahead of better single-shot accuracy:
- Environment grounding: deterministic filesystem and tool access, so an agent reliably locates and writes the right file instead of guessing across a simulated Workspace and Box layout.
- Explicit ambiguity resolution: resolve underspecified instructions before LLM reasoning commits, rather than letting the model improvise a plausible-but-wrong interpretation.
- Cross-document state tracking: hold context across multiple domains and platforms over long horizons, the exact axis where APEX models struggle most .
Two caveats before treating these scores as settled. Both benchmarks carry vendor interest (Mercor sells training data, Scale AI sells evaluation services), and independent replication on the private test sets has not happened yet, so read the numbers as directionally correct, not final. And the trajectory is steep: APEX scores reportedly climbed from 5-10% a year earlier to 20%+ now , which means static capability assumptions will likely be wrong within quarters.
The gap to autonomous delivery is real, but so is the rate of closure. Build for consistency across runs now, and re-benchmark every quarter; the floor these tests measure is moving up faster than any single headline score suggests.
Frequently asked questions
What does APEX-Agents measure that standard AI benchmarks miss?
APEX-Agents measures long-horizon, multi-step professional work, not single-prompt trivia or isolated coding puzzles. Mercor places agents inside a simulated Google Workspace with a Box file system and scores each run against 1-10 expert-written, client-ready pass/fail criteria in investment banking, consulting, and corporate law . The point is workflow competence: finding the right file, holding context, handling ambiguity. Leaderboard quiz scores never test any of that.
Why is the RLI autonomous rate (2.5%) so much lower than APEX's (24%)?
Scope. The Remote Labor Index uses 240 real commissioned freelance projects with heterogeneous file types, ambiguous deliverable standards, and individual completion times exceeding 100 hours . The best agent, Manus, automated only 2.5% of them . APEX measures single-session task pass rates, where Gemini 3 Flash reached ~24% . RLI is the harder, longer-horizon test of finishing a whole project.
Does GDPval's 47.6% contradict the sub-25% APEX and RLI rates?
No. They measure different claims. GDPval tests output quality on specified, single-deliverable occupational tasks; Claude Opus 4.1 matched or beat the human reference on 47.6% of its gold subset . That is augmentation. APEX and RLI test autonomous project completion from ambiguous brief to final deliverable. That is automation. A model can produce strong specified outputs while still failing to finish whole commissioned projects, so both results hold at once.
Which model performed best on APEX-Agents, and by how much?
Gemini 3 Flash (Thinking=High) led at ~24%, with GPT-5.2 at ~23% . Claude Opus 4.5, Gemini 3 Pro, and GPT-5 clustered near 18%, and the top agent reached about 40% only when granted eight attempts . The spread between models is narrower than the gap between every model and client-acceptable performance.
Are APEX-Agents results independent of the vendors who built them?
Not fully. Mercor, a training-data vendor, built APEX, and Scale AI, an evaluation and data vendor, co-produced RLI . Both have commercial reasons to highlight capability gaps. Mercor's evaluation infrastructure (Archipelago) is open source, but private test sets (RLI keeps 230 of 240 projects held back) prevent full external replication . Treat the findings as directional until independent replication exists.
