Semantic Search, Hybrid Search, and Reranking in AI Agents

Every capable agent is, underneath, a machine for finding the right few things in a large pile of possibilities: the right documents to answer from, the right past conversation to remember, the right tool to call, the right cached answer to reuse. All four are the same problem wearing different clothes (retrieval), and the quality of that retrieval sets a ceiling the smartest base model can't lift. Feed an agent the wrong three chunks and it will fluently, confidently answer the wrong question. So it's worth being precise about the three techniques that dominate modern retrieval (semantic search, hybrid search, and reranking) and where in an agent they actually earn their cost.

The framing we keep returning to is that an AI agent is a backend microservice with an inference layer, and retrieval is its database access pattern. The twist is that the query is now natural language and "matching" means matching meaning, not strings. That single shift is what makes semantic search necessary, and what makes it fail in ways that force you into hybrid search and reranking to get back the precision you gave up.

Semantic Search: Matching Meaning, Not Words

Classic keyword search matches tokens. Ask it for "how do I reset my password" and it hunts for the literal words reset and password; a document titled "recovering account access" (a perfect answer) scores zero. Semantic search fixes this by mapping both the query and every document into a high-dimensional vector space with an embedding model, so that things which mean the same land near each other. "Reset my password" and "recover account access" become nearby points; retrieval is a nearest-neighbor lookup, ranked by cosine similarity.

The embedding model here is a bi-encoder: it encodes the query and each document independently into fixed vectors. That independence is the entire reason it scales. You embed your whole corpus once, offline, store the vectors in a vector index, and at query time you only embed the query and do an approximate nearest-neighbor (ANN) search: millions of documents in single-digit milliseconds. The document vectors never need to know the query exists. It's the same trick as a database index: precompute the expensive part, keep the hot path cheap.

What you get is genuine robustness to paraphrase, synonym, and intent. What you give up is exactness, and the failure modes are specific and unglamorous:

• Exact-match terms it fudges. Product SKUs, error codes, order IDs, function names, a specific person's surname: an embedding of ERR_2043 sits suspiciously close to ERR_2049, and "the Apollo contract" may retrieve every contract that feels vaguely important.
• Rare and out-of-vocabulary tokens. A term the embedding model never saw in training has no meaningful place in the space; it can't encode what it doesn't understand.
• Negation and precise logic. "invoices not yet paid" and "invoices paid" embed almost identically; the semantic gist is the same even though the meaning is opposite.

None of these are bugs to be patched out. They're the structural cost of compressing meaning into a single vector. Which is exactly why the next technique exists.

Hybrid Search: Two Retrievers That Fail Differently

The insight behind hybrid search is that keyword search and semantic search fail in opposite directions, so their errors are uncorrelated. Lexical search (almost always BM25, the decades-old sparse ranking function) is unbeatable at exactly the things dense embeddings fumble: exact identifiers, rare terms, precise phrases, codes. Dense semantic search wins on paraphrase and intent. Run both, combine their results, and each covers the other's blind spot. That's the whole idea, and it's one of the highest-return-on-effort moves in all of retrieval.

The one subtlety is how you combine them, because the two retrievers speak different languages. BM25 emits unbounded relevance scores; cosine similarity lives in [-1, 1]. Averaging them directly is meaningless; you'd be adding temperatures to distances. The clean fix, now the industry default, is Reciprocal Rank Fusion (RRF), which throws away the raw scores entirely and fuses on rank position instead:

score(d) = Σ 1 / (k + rank(d)), summed across every retriever that returned document d, with a constant k (conventionally 60) damping the influence of low ranks.

Because it operates purely on ranks, RRF is immune to the incompatible-scale problem, it needs essentially no tuning, and it's about three lines of code. A document that both retrievers rank highly floats to the top; one that only lexical or only semantic search loved still gets a fair hearing. On e-commerce retrieval benchmarks a tuned hybrid setup has posted roughly 7% higher NDCG than either retriever alone, and RRF fusion has shown double-digit gains in MAP over BM25 by itself. The lesson worth internalizing: the fusion math is trivial; the real engineering is in your embedding model choice and your chunking strategy, not the algorithm.

Reranking: Precision at the Very Top

Hybrid search gives you good recall: the right answer is very likely somewhere in your top 50. But an agent doesn't get to read 50 chunks; context is finite and expensive, and burying the answer at rank 30 is nearly as useless as not retrieving it. You need precision at the top: the best 3–5, in the right order. That's the job of a reranker, and it works by spending compute you couldn't afford at corpus scale on the tiny shortlist where it matters.

The workhorse is a cross-encoder. Where the bi-encoder embedded query and document separately, a cross-encoder takes a (query, document) pair together as one input and lets every token of the query attend to every token of the document before emitting a single relevance score. That full cross-attention is dramatically more accurate (it can see that "not paid" contradicts "paid," that this ERR_2043 is the right one) precisely because it never compressed each side into an isolated vector.

The reason you can't just use a cross-encoder for everything is cost. It scores one pair at a time and shares no precomputation, so ranking a million documents means a million forward passes at query time, hopeless. Bi-encoders scale and blur; cross-encoders are precise and don't scale. So you compose them into the pattern that defines modern retrieval:

Retrieve wide and cheap, then rerank narrow and expensive. In production RAG pipelines a cross-encoder reranking stage commonly lifts NDCG@10 by 5–15 points (more on lexically hard data) for well under 200 ms of added latency, because it's only scoring a few dozen pairs, not a corpus. Managed rerankers like Cohere Rerank and small open models like ms-marco-MiniLM-L-6-v2 make this a single API call or a lightweight local model. This three-stage shape (hybrid retrieve, fuse, cross-encode) is the reference architecture, and everything below is an application of it.

The mental model that ties it together: bi-encoders are a coarse filter that scales to millions, cross-encoders are a fine filter that scales to dozens, and hybrid search makes the coarse filter's recall trustworthy enough that the fine filter has good raw material to work with. Miss any stage and you feel it: poor recall, or a right answer stranded at rank 30, or a latency bill you can't pay.

Application 1: Semantic Caching

Here's the first place this stack quietly pays for itself, and it's not RAG. Most production agent traffic is repetitive: users ask the same things in different words, and agent frameworks resend near-identical prompts constantly. A traditional cache keyed on an exact string hash catches none of it: "how do I reset my password" and "I forgot my password, help" are different bytes, so it misses. A semantic cache keys on meaning instead. It's semantic search pointed at your own history: embed each incoming query, ANN-search the store of past (query → response) pairs, and if the nearest neighbor sits above a similarity threshold, return the stored answer, skipping the model call entirely.

The upside is enormous and immediate: a cache hit costs an embedding call and a vector lookup (a few milliseconds and a fraction of a cent) instead of a multi-second, multi-thousand-token generation. Reported production hit rates land anywhere from 30% to 70% depending on how repetitive the traffic is. That's a direct, compounding cut to both latency and spend on the highest-volume path an agent has.

But semantic caching is where the innocent-looking similarity threshold turns into the whole ballgame, because a cache false positive is a wrong answer served to a user: not a slow answer, a wrong one. Set the threshold too low and the cache gets "confidently helpful": it decides "how do I cancel my subscription" is close enough to a cached "how do I upgrade my subscription" and serves the wrong policy. Set it too high and hit rates collapse and you've built an expensive no-op.

The threshold isn't a universal constant either; it's specific to your embedding model and your domain. Empirically, optimal cutoffs land in very different places for different encoders (~0.83 for one, ~0.78 for another) with no way to know but measurement. So the non-negotiable engineering practice is an eval loop: sample 1–5% of cache hits and blind-grade the cached answer against what the live model would have said, with a human or an LLM-as-a-judge. A semantic cache without that loop is a footgun; it silently trades correctness for savings and you won't see the bill until users do.

And this is where reranking re-enters through the side door. The strongest semantic caches don't trust the raw ANN similarity as the final word: they retrieve the top few candidate cache entries and run a cheap cross-encoder verification to confirm the cached query genuinely means the same thing as the new one before serving: the exact same "coarse filter, then fine filter" pattern, now guarding correctness instead of relevance. Start conservative, add the verifier, and let the eval loop earn every basis point of threshold you loosen.

Application 2: Reranking for Tool Selection

The second high-leverage application is one many teams don't recognize as retrieval at all, because it feels like an agent-design problem. As you connect an agent to more tools (and with MCP, "more" quickly means dozens or hundreds across many servers) you hit a hard wall. Every tool's name, description, and schema gets stuffed into the context window on every single call, and two things break at once: the prompt bloats (cost and latency climb with each tool you add), and the model's accuracy at picking the right tool degrades. Benchmarks show tool-selection accuracy falling off once you pass roughly 10–15 tools. More capability makes the agent worse. Tool retrieval is now the primary bottleneck in large MCP settings.

The fix is to stop treating the tool list as static context and start treating it as a retrieval corpus. Don't show the agent every tool; retrieve the handful relevant to the current step and show only those. This is precisely the three-stage stack again, pointed at tool descriptions instead of documents:

1. Retrieve: embed the user's intent (or the current sub-goal) and hybrid-search it against an index of all tool descriptions to pull, say, the top 10–20 candidate tools.
2. Rerank: a cross-encoder (sometimes called a tool refiner) scores those candidates against the query together, understanding the fine distinctions between similar tools (three different "send email" tools with different auth scopes) that a bi-encoder blurs, and cuts to the top 3–5.
3. Present: only those finalists go into the prompt for the model to actually choose from and call.

The numbers are striking. The RAG-MCP work (2025) reported that retrieval-based tool selection more than tripled selection accuracy (from about 13.6% to 43.1% on their benchmark) while cutting prompt tokens by more than half. You make the agent cheaper and smarter at the same time, and the mechanism is nothing more exotic than the retrieval pipeline you already understand. The same move generalizes: reranking is how you pick the right few memories to load, the right few examples for a dynamic prompt, the right sub-agent to route to. Anywhere an agent faces "too many options to show them all," the answer is retrieve wide, rerank narrow.

The reframe worth carrying out of this: "which tool should the agent use?" and "which documents answer this question?" and "have I answered this before?" are the same query against different corpora. Once you see tool selection and cache lookup as retrieval problems, the three-stage stack you built for RAG is already the solution; you just point it somewhere new.

What Actually Bites in Production

The concepts are clean; the failures live in the plumbing. A field checklist:

• Chunking dominates retrieval quality. More than embedding-model choice, how you split documents decides whether the answer is even retrievable. Chunks too big dilute the signal; too small sever the context. Tune this first; it's the cheapest large win.
• Always add BM25 before you tune anything fancy. Hybrid search is the highest ROI move available, and it's the one that saves you when a user pastes an exact error code or an order ID your embeddings turn to mush.
• Rerank a shortlist, never the corpus. The whole point of the two-stage split is that the expensive model only ever sees a few dozen candidates. If your reranker latency is hurting, your first-stage k is too big.
• Treat the cache similarity threshold as a safety control, not a tuning knob. A false positive here is a wrong answer to a user. Start conservative, add a cross-encoder verifier, and never loosen it without the eval loop watching.
• Measure retrieval separately from generation. When an agent answers wrong, you need to know whether it retrieved the wrong context or reasoned wrong over the right context. Without recall@k and rerank metrics you're debugging blind, the same way you'd never debug a service without separating a bad query from bad business logic.
• Watch the order and the compounding. A pipeline is a product of its stages: a narrowing filter placed ahead of a smarter one can silently eat the very candidates the smart stage was meant to rank. Test the pipeline end-to-end, not just each part.

Closing Thoughts

Semantic search, hybrid search, and reranking read like three topics and behave like one: a coarse retriever that scales to millions with acceptable blur, a lexical partner that covers its exact-match blind spot, and a fine reranker that spends real compute on the short list where precision decides the outcome. Retrieve wide, fuse, rerank narrow. Learn the shape once and you stop seeing separate techniques and start seeing a single dial between recall and precision that you set stage by stage.

The part worth remembering is that this stack isn't only for the obvious RAG use case. The two places it moves the needle hardest inside an agent are the ones that don't look like search at first glance: caching, where it turns repetitive traffic into near-free responses, and tool selection, where it turns an unusable pile of a hundred tools into the right three. Which lands us, as it always seems to, on the same note: a reliable agent is an architecture achievement, not a model one. A stronger base model won't fix bad chunking, won't set your cache threshold, and won't decide which tools the agent gets to see. Those are retrieval decisions, and retrieval is where good agents are quietly won or lost.

References & further reading:
Reciprocal Rank Fusion, explained · Hybrid Search for RAG: Combining BM25 and Dense Vectors · Hybrid Search: BM25, Vector & Reranking Reference · RAG Reranking with Cross-Encoders · GPT Semantic Cache: Reducing LLM Cost and Latency · Portkey: Semantic Caching Thresholds and Why They Matter · RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection · Dynamic ReAct: Scalable Tool Selection for Large MCP Environments

← Back to all posts