💡 TL;DR & Key Takeaways
A new paper demonstrates that adding 1,000 irrelevant skills to an LLM agent's library causes loaded-skill recall to plummet from 73.6% to 27.4% and the zero-load rate to spike to 51.6%, primarily because agents fail to load any skill rather than selecting the wrong one. This "implicit routing" failure is driven by semantic interference from distractor descriptions during execution, which can be mitigated by masking descriptions or using explicit pre-selection methods, though the latter introduces latency and potential omission risks.
Failure Mode: Agents suffer from under-loading rather than substitution, with only 2.32% of evaluations loading non-golden skills, meaning task success metrics like Pass@3 often mask the underlying routing breakdown.
Mitigation Strategies: Masking irrelevant skill descriptions restores most loading capability by reducing semantic interference, while explicit pre-selection (via LLM, RAG, or hybrid methods) improves recall at the cost of increased routing time and the risk of omitting necessary skills.
Practical Implications: The author recommends reporting recall and zero-load rates alongside task success, writing descriptions that clarify what a skill is not for, and maintaining a hand-written index to enforce explicit pre-selection, noting that even small libraries can face these issues if plugins expand the prompt context significantly
A new paper measured what happens when you bury the right skill under a thousand irrelevant ones. The agent does not pick a wrong skill. It mostly stops loading any.
In "Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale", adding 1,000 irrelevant skills to an agent's library cut loaded-skill recall from 73.6% to 27.4% and raised the share of runs that load no skill from 9.3% to 51.6%, while task success fell far less. Agents mostly load nothing rather than the wrong skill, so Pass@3 alone hides the failure.
Relevant skills stay installed and recognizable: when the same models were asked only to select skills, recall stayed at 73.9%.
The loss comes from routing becoming a side task during execution, not from the agent confusing one skill for another.
Masking only the descriptions of the irrelevant skills restored most of the loading.
My own 20-skill library is far below the paper's tested range, so it shows nothing about collapse at my size.
These are field notes, not a replication. The numbers are the paper's, from its full text. What I measured myself is small, and I say what it is. I also tried to run a routing probe and could not finish it, and I explain that below.
Why does skill routing matter?
Agent skills use progressive disclosure. The model first sees only each skill's name and description, and opens the full file only if it judges the skill relevant. The paper calls that decision implicit routing: the agent decides whether and which skills to load while it is also planning, reading files and solving the task.
It keeps long skill files out of the context. It also assumes the agent can pick the few right descriptions out of many while doing something else. That assumption is what the paper tests.
I wrote about the moving parts around this in Prompt, Context, Harness, Loop: An Agent's Anatomy, and about packaging operational knowledge for coding agents in Repo-to-Skill Framework Boosts Agent Performance. This paper asks the question those posts leave open: does the agent find the skill when there are many?
Under implicit routing the agent sees every skill description and decides while working; under explicit pre-selection a filter shows it at most five skills first.
What did the paper test?
Item | Value |
|---|---|
Benchmark | SkillsBench: fixed tasks, workspaces, verifiers, golden skills |
Tasks | 82 of 87 (5 excluded for infrastructure problems) |
Variable | number of added irrelevant skills: 0, 10, 100, 1,000, 10,000 |
Distractors | sampled from a downloaded pool of more than 20,000 skills, fixed seed |
Main models | GLM-5, Kimi K2.5, MiniMax M2.5, 3 runs each |
Scale | more than 15,000 trajectories |
Check on stronger models | Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.4 on a 37-task subset |
Only the number of added skills changes. The task, the verifier and the golden skills stay fixed.
What happened as irrelevant skills grew?
Added irrelevant skills | Loaded-skill recall | Zero-load rate | Pass@3 |
|---|---|---|---|
0 | 73.6% | 9.3% | 51.6% |
1,000 | 27.4% | 51.6% | 38.6% |
Explicit selection from the same 1,000-skill library kept 73.9% recall. Source: abstract and Section 1 of the paper. Table 1 reports 9.2% and 50.8% for zero-load because it covers a slightly broader task set (86 tasks).
The gap between recall and task success is the useful part. At 100 added skills, recall already dropped by 24.6, 15.6 and 20.7 points for the three open-weight models, while GLM-5's Pass@3 barely moved (57.3% to 56.1%). On the 37-task subset, Claude Opus 4.6's Pass@3 went from 64.9% to 62.2% at 1,000 skills while its recall fell from 81.3% to 30.9%.
Strong agents solve around a missing skill. The task still passes, and the skill system is quietly broken. The paper's advice is to report recall and zero-load rate next to task success.
Is it the wrong skill or no skill?
No skill. Loading a non-golden skill happened in 2.32% of evaluations, and only 0.95% loaded something similar to a missed golden skill (cosine similarity above 0.8, such as pdf-reader instead of pdf). Under-loading, not substitution, explains the decline.
What causes the collapse?
The paper ran two interventions on a 10-task subset at 1,000 irrelevant skills:
Setting | Loaded-skill recall | Zero-load rate |
|---|---|---|
No extra skills (reference) | 73.5% | 5.6% |
1,000 irrelevant skills, original | 18.6% | 62.2% |
Descriptions masked with placeholders | 63.9% | 23.6% |
Plan-first prompt, metadata intact | 44.6% | 15.7% |
LLM preselection | 50.4% | 10.0% |
RAG preselection | 56.4% | 21.6% |
Hybrid preselection (retrieve, then LLM picks) | 61.1% | 11.1% |
Masking keeps the same number of slots and similar length, so context length alone does not explain the loss. Masking only the descriptions restored recall to 63.3%.
The authors read this as semantic interference from distractor descriptions, plus instruction interference when choosing a skill competes with doing the task. That is an inference from a 10-task subset, so I hold it loosely. The plan-first prompt helped, but less than selecting before execution.
Preselection has a price. Median routing time rose from 8 to 19 seconds for LLM preselection and from 33 to 41 seconds for hybrid, going from 10 to 1,000 distractors. It can also omit a skill the task needs later.
Does this apply at 20 skills?
The paper does not say. Its smallest non-zero library adds 10 skills, and the figures I have are for 100 and 1,000 added skills, where the drop is large. My skill suite has 20 skills in total.
Two things stop me from shrugging. First, a coding agent with plugins loaded can list far more than 20 skills in its prompt. Mine runs well past a hundred entries. Second, my repo does not rely on implicit routing at all. AGENT.md carries a stage-to-skill table, and CLAUDE.md tells the agent to read a stage's SKILL.md in full before acting. That is explicit preselection by hand, which is the paper's mitigation, done in prose.
What did I try to measure, and why is there no number?
I wrote a probe. Thirty tasks, each with a skill I judged correct from CLAUDE.md. The library was my 20 skills plus up to 693 real skills from other folders on my machine. One mode was a small agent loop with a load_skill tool and a stub shell tool, scoring whether the model loads a skill within three turns. The other asked the model directly to pick skills.
A small pilot ran on free models. A larger run then hit the free tier's cap of 50 requests per day before I had a usable dataset. I am reporting no routing numbers from it.
Two honest limits would have applied anyway. The task labels are mine, not independent ground truth. And a three-turn loop with a stubbed shell is a proxy for the paper's real execution environment, not a replication of it. If I run it properly, I will say so and publish the results then.
Rules I am taking from it
My opinion, not a measurement:
Report loaded-skill recall and zero-load rate. Task success hides routing failure.
Write descriptions that say what the skill is not for, and where to go instead.
Keep a hand-written index of which skill belongs to which stage, and make the agent read it first.
Log which skills actually get loaded. If a skill never loads, you will not find out from the output.
Treat every skill you add as a cost to all the others.
For the harness-versus-framework distinction, see Agent harness or agent framework? They're not the same.
How to check this yourself
Paper: arXiv 2610.18522, "Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale".
The authors link code at
github.com/jinlinll/lost-in-implicit-routing.
If you have a skill library over 100 and you log which skills load, I would like to know what your zero-load rate is.
Share your thoughts in the comments — I'd love to hear how this technology is impacting your industry.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!