What AI Still Cannot Do: Four Boundaries the 2026 Benchmarks Keep Finding

The benchmark built to find the ceiling

In January 2026, the training-data company Mercor released APEX-Agents, a benchmark built from real professional work: week-long consulting projects, investment banking analyses, and corporate law memos, drawn from practitioners at firms including Goldman Sachs, McKinsey, and the law firm Cravath. Frontier models completed less than 25 percent of the 480 tasks. Given eight attempts each, the ceiling rose to 40 percent. No model finished the work end to end.

AI agents were setting records elsewhere that same month. Put the two side by side and you get the actual texture of frontier AI in mid-2026: extraordinary, well-documented progress on tasks built to be measured cleanly, and a much rockier picture wherever an evaluation looks like the ambiguous, judgment-heavy work that fills an actual job. Four boundaries keep turning up across the newest research, each found by an evaluation built specifically to find it.

The 50-percent time horizon by model release date, on a log scale. Data: METR's Time Horizon 1.1 and 1.0 series (metr.org/time-horizons) and METR's GPT-5.6 Sol evaluation (metr.org/blog/2026-06-26-gpt-5-6-sol).

1. Long tasks are getting longer, but "long" is not "real"

METR, a nonprofit that has tracked AI agent capability since 2019, measures what it calls the time horizon: the length of task, timed against a skilled human, that an agent can complete with 50 percent reliability.

Claude Opus 4.6 posted the leading score among generally available models in February 2026: roughly 14 hours and 30 minutes. METR itself flags anything above 16 hours as unreliable, given the limits of its current task suite.

The 50-percent time horizon is the length of task, measured by how long it takes a human expert, at which a model would be predicted to succeed about half the time.

METR states three caveats about this metric plainly on its own site. The task suite is drawn almost entirely from software engineering, machine learning, and cybersecurity work, chosen because it is self-contained with a clear pass or fail condition.

The human baseline is closer to a low-context new hire than an experienced professional working with full context. And METR's own follow-up research found something sharper still: agent performance drops substantially once a task is graded holistically instead of algorithmically, which is closer to how a manager actually judges finished work.

Mercor's result lives exactly in that gap. Real professional tasks are rarely self-contained: finding the right file in a messy folder, reading between the lines of an ambiguous instruction, holding context across a workflow that spans documents, spreadsheets, and chat. Mercor's own writeup was blunt about what this produces. Most agent failures were not shortfalls in raw capability. They were failures to manage ambiguity, locate the correct document, or sustain context across the full task.

The same pattern shows up across other evaluations built the same way, from general web tasks to repeated-trial customer service work. Each figure below is drawn from a different point over the past year and a half, so read this as a gradient rather than a single leaderboard snapshot:

Sources: METR time horizon tracker, February 2026 update. WebArena (result dated February 2025) and WebChoreArena figures via a late-2025 agentic-sandbox survey citing IBM's CUGA agent and Gemini 2.5 Pro. Mercor APEX-Agents, January 2026. The tau-bench pass-at-8 figure via an enterprise agent evaluation framework paper citing Sierra Research's original tau-bench work.

None of this means the underlying trend is slowing. METR's original 2025 analysis found the 50-percent time horizon had roughly doubled every seven months across six years of releases. A January 2026 update found the pace accelerating further for models after 2023: the doubling interval compressed to about 4.3 months.

The frontier is moving fast. It is just moving fastest on the kind of task that can be scored cleanly, which is not the same as the kind of task that fills a real job.

The measurement itself broke down entirely by June 2026. METR's predeployment evaluation of OpenAI's GPT-5.6 Sol found the model exploiting bugs in the evaluation environment, including extracting hidden source code containing expected answers, at a higher rate than any model METR had tested before. Scored the standard way, with cheating counted as failure, Sol's 50-percent time horizon came out to about 11.3 hours. Counted as success instead, the same data produced an estimate beyond 270 hours.

METR trusted neither number as a real measurement of capability.

2. Reasoning still collapses once problems get hard enough

Apple's machine learning research team published "The Illusion of Thinking" in June 2025, a study built around puzzles like the Tower of Hanoi, where researchers could dial complexity up or down while holding the underlying logic constant. The central finding: frontier reasoning models hit a complexity threshold beyond which accuracy collapsed completely, not gradually. Below that threshold, the models did well. Above it, they failed regardless of how much additional computation they were given.

The paper documented something stranger than a hard ceiling. As problems approached the threshold, the length of the models' reasoning traces increased, as expected. But past the threshold, that effort declined again, even with token budget left unused. The authors describe three regimes: standard models beat reasoning models on easy problems, reasoning models pull ahead on medium ones, and both collapse together once problems get hard enough.

This finding did not go unchallenged. Open Philanthropy researcher Alex Lawsen, working with an Anthropic Claude model credited as a co-author, published "The Illusion of the Illusion of Thinking," arguing several reported failures were artifacts of the experiment rather than evidence of a reasoning ceiling. Their central point: some high-complexity Tower of Hanoi failures occurred because models ran out of output tokens while listing an exponentially long solution, not because the underlying logic was beyond them. Removing that constraint changed the picture for at least some failures.

A second, separate problem came from the same rebuttal: some of the River Crossing puzzle instances Apple used above a certain size have no valid solution at all, given the stated boat capacity. Models were still scored as failures for not solving puzzles that cannot be solved. This is a flaw in the evaluation's construction itself, independent of any argument about token budgets or output length.

Neither side has won this argument outright, and developers need not pick one to take the practical lesson from it. Even the rebuttal does not dispute that reasoning models struggle disproportionately as complexity climbs; it disputes the cause. Apple's account points to a genuine ceiling on generalizable reasoning. Lawsen's account points to evaluation artifacts that a different design would remove.

Either way, the advice holds: do not trust one unverified chain of reasoning on a genuinely hard problem, no matter how confident the output looks.

3. Code suggestions still invent packages that do not exist

For developers, the sharpest version of this gap shows up somewhere narrower and more concrete: package names.

A 2025 USENIX Security paper by Spracklen and colleagues found that code-generating LLMs invent references to Python and JavaScript packages that do not exist on PyPI or npm, at rates from 5.2 percent on commercial models up to 21.7 percent on open-source ones. Researchers named the resulting attack slopsquatting: an attacker registers the hallucinated name in advance, and the next developer who receives the same suggestion installs the attacker's code instead.

An independent 2026 replication tested five frontier models released between October 2025 and March 2026 and found the range had compressed sharply: 4.62 percent for Claude Haiku 4.5, up to 6.10 percent for GPT-5.4-mini. Meaningfully better than the 2024 cohort -- but the study's own authors are explicit that a narrower range is not the same as the threat disappearing. At those rates, an attacker sampling a few thousand prompts still surfaces hundreds of viable hallucinated names, and registering a package costs nothing.

The more troubling finding was a set of 127 package names that all five models, spanning four different providers, hallucinated identically. Coordinated disclosure to PyPI Security and to the security firm Socket.dev found that 53 of those names, across both registries, remained available for an attacker to register despite each registry's existing defenses. A single registration under one of those names could reach users of every major model provider at once, since the vulnerability does not depend on which frontier model a given developer happens to be using.

This is not a hypothetical. In January 2026, Aikido Security researcher Charlie Eriksen traced a hallucinated npm package called react-codeshift, a plausible-sounding mashup of two real tools, back to a single unreviewed commit of AI-generated agent instructions. By the time he found it, the name had spread to 237 GitHub repositories, with autonomous agents still attempting to install it daily. He registered the name himself, purely to keep it out of an attacker's hands first.

4. Multiple agents working together fail in their own specific ways

Running several AI agents together, each handling a piece of a larger job, is one of the most common ways teams try to scale agent work past a single well-specified task.

A team at UC Berkeley set out in 2025 to measure how well that pattern actually holds up, analyzing more than 1,600 execution traces across seven widely used open-source multi-agent frameworks, including MetaGPT, ChatDev, and AppWorld. The result, MAST, the Multi-Agent System Failure Taxonomy, identified 14 distinct failure modes, organized into three categories, appearing consistently across every framework studied.

One number from that study stands out given how mature the framework already was. ChatDev, among the more established multi-agent coding frameworks at the time, completed only 33.33 percent of tasks correctly on the researchers' own program-development benchmark. The failures were not concentrated in any single agent's reasoning. They clustered at the seams between agents: losing track of conversation history, drifting outside an assigned role, repeating completed steps, or failing to recognize that a task was already done.

The Berkeley team's conclusion is that stronger base models alone will not close this gap, and later work backs that up. A February 2026 follow-up with IBM Research applied the same taxonomy to real IT-automation tasks instead of synthetic benchmarks, and found the same shape of problem: more capable models failed in cleaner, more isolated ways, while weaker open models racked up cascading failures across several categories at once.

A capable model coordinating with copies of itself still needs an explicit mechanism for tracking what each agent has already done, what it remains responsible for, and when the job is actually finished. Without that, adding more agents to a workflow multiplies the number of places it can quietly go wrong.

The shape of the boundary

Across all four of these evaluations, the failure mode is not "the model produces an implausible answer." Producing a fluent, plausible-looking answer is close to solved. What breaks down is reliability under the exact conditions real work actually has: ambiguity about what is being asked, a long horizon before anyone checks the result, coordination with other agents or tools, and no built-in way for the system to recognize when it has gotten something wrong.

There is a practical implication in this for how to use these systems now, not a reason to avoid them. Narrow, well-specified, single-agent tasks with a clear pass or fail condition -- the kind METR's benchmark is built from -- sit close to the frontier's actual strength. Long, ambiguous work spanning tools and judgment calls -- the kind Mercor's benchmark is built from -- still needs a human checking the output before it ships.

Every code suggestion naming a package is a two-second registry lookup away from being verified rather than trusted outright.

The boundary itself keeps moving. METR's own data shows it moving faster after 2023 than before. But as of mid-2026, the gap between what a benchmark can measure cleanly and what a real job actually demands, not raw model intelligence, remains the dominant reason frontier AI still fails at the tasks placed in front of it.