# k-onfusion: 200,000 tools, and the agent picks the wrong one > I gave an AI agent 200,000 tools and it went from solving 85% of tasks to 30%, mostly by calling a lookalike of the right tool. Javed Habib · Sep 27, 2026 · #llm #agents #mcp #retrieval #claude-code #experiment #jev https://jaeaeich.com/writing/k-onfusion Imagine walking into a restaurant and the waiter handing you a menu the size of an encyclopedia. I'd be overwhelmed, and a little worried too, because why does one kitchen need 27 kinds of chicken? Agents have the same problem with MCP tools: give one 10 tools and it can read all of them and pick the right one (and if it picks wrong, it just tries another), but that stops working once the menu has 200,000 things on it. I wanted to find out where it stops working and why, so I built [k-onfused](https://github.com/jaeaeich/k-onfused), a small test bench that hands an agent a task and a catalog of up to 200,000 tools and checks whether it called the right tool with the right arguments. Groceries are expensive, so the whole thing ran on [headless Claude Code](https://code.claude.com/docs/en/headless) with my existing subscription, which cost me some evenings and a big bite of my usage window, and nothing on an API bill. ## Results up front - **60%** tasks solved at 200k tools, 10 per search (from 85% at 100 tools) - **30%** tasks solved at 200k tools, 3 per search (from 85% at 100 tools) - **67%** right tool picked once it was in the results, 200k, 10 per search (from 85% at 100 tools) Up to 10,000 tools, the agent barely notices how big the catalog is, and somewhere between there and 200,000 it **falls off a cliff**. When the search hands back only 3 tools, the right one often never reaches the agent at all, and when it hands back 10, the right one is usually there but the agent picks its lookalike, a `v2` of the same tool or the by-id version when it was given a URL. I tried the 2 obvious fixes, a second agent that does the searching for it and a small fast model that re-sorts the search results, and **neither moved the number** by more than noise, because at that size the search is mostly doing its job and what's left is the agent choosing between near-identical tools it was already handed. ## Building a 200,000-tool catalog Deploying 200,000 real tools was never going to happen, but I did need 200,000 of them, so I mocked them. They're [generated from a seed](https://github.com/jaeaeich/k-onfused/blob/main/src/gen/tools.ts)[^seed] so every run sees the same catalog: - apps in several editions (Slack, Slack Enterprise, Slack EU) - the things you'd expect inside them like issues, invoices and job postings - the usual actions on each Playing devil's advocate, I also made the catalog confusing on purpose, the way real ones get once a few vendors ship a `v2` and never retire the `v1`, so there's a `daybay_create_event` next to a `daybay_v2_create_event`, and a by-id and a by-url version of most actions. The [tasks](https://github.com/jaeaeich/k-onfused/blob/main/src/gen/tasks.ts) never reuse a tool's wording either, so where the tool says *"Permanently removes a job posting in Staffwise, identified by its numeric ID"*, the task says *"Nuke vacancy number 28286 in Staffwise"*, because "nuke" is what a person would actually type and I wanted the agent to make that jump instead of matching words. ## How the agent searches Pushing 200,000 tool definitions straight into the model's context isn't really an option, unless you enjoy watching your credits disappear, so the agent needs a way to ask for tools instead. The idea came from the encyclopedia again: when I want to read about an animal I don't flip through the whole thing; I go to the animals section. I put every tool into [Qdrant](https://qdrant.tech), a database that searches by meaning rather than exact words,[^local] so when the agent asks for something about GitHub it gets back the tools closest to what it asked for. How many tools come back is called k (hence the name), and it's the knob this whole post turns, since a bigger k gives the agent more to choose from and a smaller one gives it less. Every trial starts the agent with a single tool, [`request_tools`](https://github.com/jaeaeich/k-onfused/blob/main/src/agents/session-worker.ts#L12), where it describes what it needs in its own words, gets k tools back and has to call the right one: task → request_tools → search → top k → call → grade *Figure: One trial, where k is how many tools the search hands back.* A trial only passes if the right tool got called with the right arguments.[^ajv] I ran 20 tasks (15 that need 1 call and 5 that need 2) against catalogs of 100, 10,000 and 200,000 tools, with either 3 or 10 tools per search, all on [Claude Haiku 4.5](https://www.anthropic.com/news/claude-haiku-4-5). The agent runs inside Claude Code with [everything else switched off](https://github.com/jaeaeich/k-onfused/blob/main/src/cc/headless.ts#L88), because the last thing I wanted was my own `CLAUDE.md` quietly helping it in every trial, and every trial records what it did so I could open any one of them in [Arize Phoenix](https://phoenix.arize.com) and see where it went wrong, which is how I can tell a tool that never showed up apart from a tool the agent walked right past. ## Where accuracy falls off Going from 100 tools to 10,000 changed almost nothing, and going to 200,000 cost a quarter of the tasks with 10 per search and **more than half** of them with 3: *Figure: Tasks solved as the catalog grows, 20 tasks per point.* With 3 tools per search, **11 of the 17 tasks** that passed at 100 tools failed at 200,000, and not a single one went the other way, which, with only 20 tasks, makes it the one result in this post I'm confident isn't noise. ## Why lookalikes win It breaks in 2 different ways, depending on how many tools come back. **With 3 per search, the right tool mostly never shows up.** It reached the agent in only 45% of trials at 200,000 tools, and although the agent noticed and searched again with different words, there were simply too many near-matches ahead of it in line. **With 10 per search, it shows up and the agent picks its twin.** The right tool was handed over 90% of the time, but when it was, the agent called it only 2 times out of 3. My favourite example is a task that asked it to *"Set up an appointment onto Team Calendar on Daybay"*, where the right tool, `daybay_create_event`, came back first out of 10, and the agent went down the list to number 6, called `daybay_v2_create_event` with perfectly valid arguments and then told me it had successfully set up the appointment. It made the **same choice every single time** at 10,000 tools and above, which I honestly find hard to blame it for, and the other misses look the same, either another edition of the right app or the by-id version of a tool when the prompt gave a URL. *Figure: Two real trials at 200,000 tools: the agent's own searches, what came back and what it called. Both times it reported success.* ## Two fixes that didn't help If the agent gets k tools back and still has to decide by itself, the natural move is to put something smarter in between, and I tried one slow version of that and one fast one, both only at 200,000 tools since that's where the problem was. ### The librarian Now think about another thing: what if the agent didn't search *at all*? The librarian is [a second Claude session](https://github.com/jaeaeich/k-onfused/blob/main/src/agents/session-query.ts) that the main agent (the worker) goes to and says "hey, this is the task I need to do", and the librarian runs its own searches, with filters for app, resource and action, and decides which tools the worker should get. It cost about 2.5× the model calls and time per trial and didn't clearly help, solving 45% of tasks where plain search solved 30% with 3 per search, and 55% where plain search solved 60% with 10. The traces showed why: it used a filter in only **19 of its 219 searches** even though filtering was the whole point of having it, and it usually handed back fewer tools than it was allowed to, so a bigger k was mostly wasted on it. ### The re-ranker Picking one tool out of a handful is more of a quick, instinctive call than a deep think (a [System 1](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow) thing rather than System 2), so the other idea was to skip the second LLM and re-sort the search results with a small, fast model built for that job (a re-ranker). It looked promising, because the right tool was almost always somewhere in the top 50 results but only made the top 3 about 60% of the time, so the search was finding it and just ranking it too low. I tried 3 [re-rankers](https://github.com/jaeaeich/k-onfused/blob/main/src/index/rerank.ts), and the smallest one, [MiniLM](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2), was nearly as good as the biggest while running about 20 times faster on my laptop, pushing the right tool into the top 3 **about 76% of the time** instead of 60% (both measured offline, by replaying the same 50 searches through each ranker). I put it in front of the agent, and while it *did* get the right tool to the agent more often with 3 per search (45% to 60%), the number of tasks solved **stayed at 30% either way**, and with 10 per search it came out a little lower than plain search, losing on the same twin picks as before. *Figure: Plain search against both fixes, tasks solved at 200,000 tools with 10 per search, where the re-ranker’s 42% is 8 of 19 because one of its trials hit a harness error and was dropped.* I also wanted to try [Jev by TypeSafe](https://docs.typesafe.ai), a System 1 model built for exactly this kind of pick-one-of-these decision, which might well have changed the result, but I had already burned through too many Claude Code tokens to start another round. ## Where this leaves agents Both fixes went after the search, while the traces kept pointing at the moment right after it, when the agent is looking at a `daybay_create_event` and a `daybay_v2_create_event` and has **nothing to tell them apart** but a guess. Next I'd try a stronger agent, or tool names and descriptions that make twins easier to tell apart, or Jev, once my usage window forgives me. > **lesson** > > A retrieval number can go up while the thing you care about stays flat, so > measure the task end to end before you trust a better-looking search. ## Caveats This is one model (Haiku, inside Claude Code's own agent loop), 20 tasks, one run of each and synthetic tools, so trust the shape of the curves more than the exact percentages. The drop as the catalog grows holds up [statistically](https://en.wikipedia.org/wiki/McNemar%27s_test), but the differences between plain search, the librarian and the re-ranker are **small enough to be luck**. My setup also had a bug that failed a few trials at 100 tools for reasons unrelated to the agent, so if anything the cliff is steeper than the chart shows. Every trial, every failure with a one-line reason and the exact commands to rerun it are in [the write-up for this run](https://github.com/jaeaeich/k-onfused/blob/main/results/main1/README.md), so if you think I've read something wrong, the evidence is right there. The code is at [jaeaeich/k-onfused](https://github.com/jaeaeich/k-onfused), and if you have a tool catalog this big in real life, or you get Jev running on it before I do, I'd love to hear how it went. [^seed]: The smaller catalogs are just the first 100 or 10,000 tools of the big one, so every size is a fair slice of the same menu. [^ajv]: None of the tools do anything real, so every call goes to a fake that just checks the arguments with [Ajv](https://ajv.js.org), and the right tool with a wrong argument still fails. [^local]: Tools are turned into vectors with [bge-small](https://huggingface.co/BAAI/bge-small-en-v1.5), and that model and the re-rankers further down all run on my laptop through [Transformers.js](https://huggingface.co/docs/transformers.js), so no search ever left it.