BM25 got Orama. Jev still needs one.

I built the Orama for Jev. On the first catalog it matched the best hand-tuned call and never failed a search. On real MCP tools it lost by 19 points. Both results are the argument for building it.

Some context. In the last post I tested Jev, TypeSafe AI's decision model, as a tool search for agents. It beat BM25 by 24 points. The whole search was one function around one evaluate() call.

So why would one API call need a library? It's a fair question. BM25 is twenty lines of math too, and almost nobody writes it by hand. Most people install Orama, call create, insert and search, and never think about it again. I wanted the same for Jev: a dependency you don't have to think about.

Orama packaged an algorithm. Jev needs a planner.

Orama gives you a typed schema, an index built in milliseconds, scores you can inspect and zero dependencies. It runs in the browser, at the edge and in Node, and the same query always returns the same answer.

Almost none of that survives contact with Jev. Jev is a hosted model behind a network call. It is not synchronous, not free, not offline, and not deterministic: in my repeat runs its top pick changed on about 2 of every 100 searches.

What does carry over is the shape. createIndex, insert, search, typed hits. The work moves from the algorithm to everything around the call.

A raw Jev call hides six decisions

When you call Jev yourself, you make six decisions without noticing. I measured each one on a clean benchmark: 199 tools from MetaTool, 398 requests, all arms interleaved in time.

None of these show up in a demo. Each one shows up in production.

The refusals explain two of those rows. In Jev's second week on Vercel's AI Gateway, the provider shed load on big requests: a 6.7k-token question was refused on 35 to 55% of attempts, a 1k-token one almost never. That's a capacity problem that will fade. The design lesson won't: a search library has to plan its requests, not just send them.

On a clean catalog, the best move is to ask once

Here's what surprised me. The most accurate way to use Jev on 199 tools was the simplest: one question over the whole catalog, with two identical requests racing on every attempt so a refusal costs nothing. That scored 75.9% strict.

Splitting the catalog into chunks cost about 2 points. A BM25 shortlist first cost about 5. And about two thirds of the remaining misses were defensible picks the dataset's single label called wrong, according to a blind judge that saw both tools in both orders.

So the prototype runs a cascade. Ask once, hedged. If Jev refuses twice, split the catalog into parallel chunks and run a final round over the winners. If that fails too, return the BM25 order and say so.

import { createIndex, search } from "jev-search";
 
const index = createIndex({
  documents: tools,
  id: (tool) => tool.name,
  describe: (tool) => summary(tool.description), // what the wide rounds read
  detail: (tool) => fullText(tool), // what the deciding question reads
});
 
const { hits, plan } = await search(index, { state: `User request: ${request}`, limit: 5 });
// plan: "direct" | "tournament" | "fallback", every hit carries its probability

On MetaTool it scored 75.9%, level with the best hand-tuned call. Across 398 searches it threw zero errors, and fell back to BM25 twice. I thought I was done.

On real MCP tools, the same defaults lost by 19 points

Then I ran it on the benchmark from the last post: LiveMCPBench, 525 real tools from 69 MCP servers, 94 human-annotated tasks. The comparison was the search that post shipped, a port of FastMCP's two-stage Jev search.

The prototype lost badly. 39% against 59% for the right tool first, in the same run. It also made 45 requests per search. (That search scored 56% in the last post and 54 to 59% across my runs here. With 94 tasks, a few points either way is noise. Nineteen is not.)

The cause was one assumption. MetaTool's descriptions are short: 88 characters at the median. Real MCP descriptions run from a few words to 3,800 characters. With full text in every option, a 1,600-token request held a dozen tools. The catalog shattered into dozens of chunks, and the final round compared lookalikes on the same thin text.

FastMCP had already solved this, and I had copied the wrong half. Its wide pass reads a 160-character summary of each tool. Only the final round, over eight finalists, reads the full description and the parameters. Short text to narrow, rich text to decide.

I gave the prototype that split. The gap closed to 2 points, inside the noise. Then I put Voyage embeddings in front, the setup the last post recommended:

The first-tool gap is inside the noise. The top-5 gain is not: +8.5 points, and the top five is what the agent actually sees. No search I measured on this benchmark did better there, including Voyage's reranker at 86%. It costs a quarter as much, and Jev's share of the latency is 0.64 s at the median.

The defaults are the product, so ship the eval with them

The prototype didn't get better because the algorithm got smarter. It got better because I measured it on a second catalog. Full descriptions beat short ones by 2 points on MetaTool and lost by 7 on MCP. A 1,600-token budget that was right on MetaTool shattered MCP catalogs. The same code, tuned on one catalog, would have shipped a 19-point regression to everyone with real tools.

Orama can ship defaults once, because BM25 behaves the same on every corpus. A Jev search can't. The defaults depend on your description lengths, your catalog size, and how busy the provider is this week.

So the plan for the library is short:

• The planner: summaries to narrow, details to decide, a cascade that degrades instead of failing.
• The harness: a first-class export that measures hit@1, top 5, capacity errors, latency and cost on your catalog, and picks the settings for you.

The benchmark and the prototype are open source at kachar/jev-tool-search. Measure your own catalog before you trust my defaults.

I wrote more about why owned evals matter in your evals are the moat. This is the smallest case of it I've found.

Orama made BM25 boring. The Orama for Jev won't make Jev boring. It will make it measured.