Why Cosine Similarity Fails to Catch Confusable MCP Tools

An MCP agent decides what to do based entirely on text it's handed at runtime — a tool's name, its description, and its input schema. When a server exposes multiple tools with overlapping schemas or vague boundaries, an agent can easily invoke the wrong one, or take a destructive action it wasn't meant to.

I wanted a way to catch this before it happens: flag pairs of tools that are similar enough to confuse an agent, before that server ever gets used. The obvious first approach was cosine similarity on the tool descriptions. It completely failed.

The experiment

I ran TF-IDF + cosine similarity against the 14 tools exposed by the official MCP filesystem server — 91 possible pairs. At the standard 0.85 similarity threshold, it flagged zero pairs.

Lowering the threshold didn't help — it made things worse. Genuinely confusable pairs like read_file / read_text_file and completely unrelated pairs like read_file / write_file ended up scoring in the same range. There was no threshold, anywhere, that cleanly separated "these will confuse an agent" from "these obviously won't."

Why it breaks down

Two things were going on:

1. Shared domain vocabulary swamps the signal. Tools on the same server all talk about path, file, directory — regardless of whether they're actually confusable. That overlap dominates the similarity score.

2. Opposing verbs barely move the needle. read_file and write_file share the vast majority of their tokens. The one word that actually matters — read vs. write — is a tiny fraction of the text, so it has almost no effect on a similarity score built from shared nouns.

In short: cosine similarity on tool descriptions measures topical overlap, not functional confusability. Those are different things, and the difference is exactly what matters here.

What actually worked

Instead of scoring text first, I flipped the question: can one tool's arguments satisfy the other tool's schema? If they can't, an agent can never actually confuse the two calls, no matter how similar the descriptions read. So that pair gets thrown out before any text comparison happens at all.

This structural gate — checking schema substitutability first — eliminated 63 of the 91 pairs immediately, before any similarity scoring touched them.

For the remaining 28 pairs, I scored on:

• name affinity

• non-domain token overlap (ignoring the shared path/file/directory noise)

• a hard veto on opposing verbs (read/write, create/delete)

This surfaced exactly 4 genuinely confusable pairs — the read_file / read_text_file / read_media_file cluster, and list_directory / list_directory_with_sizes — cleanly separated from everything else by a real 0.33–0.50 gap. That's the separation cosine similarity alone could never produce.

A note on scope

This is about accidental confusability between two legitimate, honestly-described tools — not malicious or poisoned tool descriptions (e.g. prompt injection hidden in metadata). That's a different, already-covered problem. This is about well-intentioned tools that are just similar enough to trip an agent up.

Try it

I packaged this as mcplock, an open-source lint check for MCP servers:

bash

pip install mcplock
mcplock lint "npx -y @modelcontextprotocol/server-filesystem ./data"

mcplock also does drift detection — it hashes tool definitions into a baseline so any runtime change to a tool's description, schema, or behavioral annotations becomes a reviewable diff, the same way a lockfile works for dependencies.

• Repo: https://github.com/yash161004/mcplock

• PyPI: https://pypi.org/project/mcplock/

If you're building or shipping MCP servers, I'd genuinely value you running it against yours and telling me what breaks.

The core takeaway: functional confusability is a structural property of tool schemas, not a text-similarity score — checking substitutability first is what actually separates risky tool pairs from safe ones.

What's your MCP server's tool count? Curious how common this problem actually is at scale.