--- title: "Why Cosine Similarity Fails to Catch Confusable MCP Tools" author: "yash161004" published: "2026-07-31T14:31:46.665Z" updated: "2026-08-28T07:05:00.292Z" url: "https://zyvop.com/why-cosine-similarity-fails-to-catch-confusable-mcp-tools-66814" canonical: "https://zyvop.com/why-cosine-similarity-fails-to-catch-confusable-mcp-tools-66814" tags: ["MCP","Schema Design","Python","open-source","developer tools","AI agents","LLM Tools","Agent Safety"] --- # Why Cosine Similarity Fails to Catch Confusable MCP Tools An MCP agent decides what to do based entirely on text it's handed at runtime — a tool's name, its description, and its input schema. When a server exposes multiple tools with overlapping schemas or vague boundaries, an agent can easily invoke the wrong one, or take a destructive action it wasn't meant to. I wanted a way to catch this before it happens: flag pairs of tools that are similar enough to confuse an agent, before that server ever gets used. The obvious first approach was cosine similarity on the tool descriptions. It completely failed. ## The experiment I ran TF-IDF + cosine similarity against the 14 tools exposed by the official MCP filesystem server — 91 possible pairs. At the standard 0.85 similarity threshold, it flagged **zero** pairs. Lowering the threshold didn't help — it made things worse. Genuinely confusable pairs like `read_file` / `read_text_file` and completely unrelated pairs like `read_file` / `write_file` ended up scoring in the same range. There was no threshold, anywhere, that cleanly separated "these will confuse an agent" from "these obviously won't." ## Why it breaks down Two things were going on: 1. **Shared domain vocabulary swamps the signal.** Tools on the same server all talk about `path`, `file`, `directory` — regardless of whether they're actually confusable. That overlap dominates the similarity score. 2. **Opposing verbs barely move the needle.** `read_file` and `write_file` share the vast majority of their tokens. The one word that actually matters — `read` vs. `write` — is a tiny fraction of the text, so it has almost no effect on a similarity score built from shared nouns. In short: cosine similarity on tool descriptions measures topical overlap, not functional confusability. Those are different things, and the difference is exactly what matters here. ## What actually worked Instead of scoring text first, I flipped the question: **can one tool's arguments satisfy the other tool's schema?** If they can't, an agent can never actually confuse the two calls, no matter how similar the descriptions read. So that pair gets thrown out before any text comparison happens at all. This structural gate — checking schema substitutability first — eliminated **63 of the 91 pairs** immediately, before any similarity scoring touched them. For the remaining 28 pairs, I scored on: - name affinity - non-domain token overlap (ignoring the shared `path`/`file`/`directory` noise) - a hard veto on opposing verbs (`read`/`write`, `create`/`delete`) This surfaced exactly **4 genuinely confusable pairs** — the `read_file` / `read_text_file` / `read_media_file` cluster, and `list_directory` / `list_directory_with_sizes` — cleanly separated from everything else by a real 0.33–0.50 gap. That's the separation cosine similarity alone could never produce. ## A note on scope This is about **accidental** confusability between two legitimate, honestly-described tools — not malicious or poisoned tool descriptions (e.g. prompt injection hidden in metadata). That's a different, already-covered problem. This is about well-intentioned tools that are just similar enough to trip an agent up. ## Try it I packaged this as `mcplock`, an open-source lint check for MCP servers: bash ```bash pip install mcplock mcplock lint "npx -y @modelcontextprotocol/server-filesystem ./data" ``` `mcplock` also does drift detection — it hashes tool definitions into a baseline so any runtime change to a tool's description, schema, or behavioral annotations becomes a reviewable diff, the same way a lockfile works for dependencies. - Repo: [https://github.com/yash161004/mcplock]() - PyPI: [https://pypi.org/project/mcplock/]() If you're building or shipping MCP servers, I'd genuinely value you running it against yours and telling me what breaks. * * * _The core takeaway: functional confusability is a structural property of tool schemas, not a text-similarity score — checking substitutability first is what actually separates risky tool pairs from safe ones._ _What's your MCP server's tool count? Curious how common this problem actually is at scale._