Hybrid Retrieval and Reranking: The Part Where We Actually Build It

Find out which half is broken before you buy anything

Two ways a retrieval pipeline fails, identical from the outside: either the right chunk was never retrieved, or it was retrieved and buried at rank 23 where the generator never saw it. Different fixes, and only one of them is a reranking problem.

The diagnostic takes an afternoon. Pull 50 real queries that produced bad answers, out of the logs, not ones you wrote to demo the system. Run retrieval at depth 50 rather than your production 5 or 8, then read the chunks yourself and mark whether one containing the answer is in there.

• No answer-bearing chunk in the 50. That is a coverage failure. Your chunking, your analyzer, or your embedding coverage is losing the document before ranking ever happens. A reranker cannot help you. It reorders the list you hand it, it cannot add to it.
• The chunk is in the 50, sitting at rank 19. That is a ranking failure, and it is what a cross encoder is actually for.

Call that first number Hit@50, or candidate coverage. It equals Recall@50 only when each query has exactly one relevant chunk, which is a convenient assumption and rarely a true one, and the two stop agreeing the moment your relevance definition allows two or three valid chunks.

Most teams I have looked at are in the first bucket and buying for the second, because the vendor dashboards show nDCG@10 and almost nobody tracks coverage. nDCG is a graded ranking quality metric comparing your ordering against the ideal one, so it is neither coverage nor precision. It does punish a query with nothing relevant in the top 10, which scores zero. What it cannot tell you is where that zero came from: a query that never had the answer in the pool and one that had it at rank 40 score identically and want opposite fixes. Averaged over a few thousand queries it will also cheerfully hide a failing slice, usually the one with the identifiers in it.⁴

Aside / the best tool is still a spreadsheet

The single best debugging tool for a bad RAG pipeline is a spreadsheet. Query in column A, retrieved chunks in column B, a human yes or no in column C. That is the whole tool, and it answered the question in an afternoon on a project where the $80K observability platform bought to do the same job was still in onboarding six weeks later. I am not against the platforms. I am against waiting for one.

General purpose embeddings are not trained to preserve the distinction that matters

Start with tokenization. Splits vary, but a common BPE vocabulary gives roughly ["SIL","10","-","9","120"] against ["SIL","10","-","9","12","k"]: the piece carrying the entire difference between ohms and kilohms is one low-information k. Some tokenizers split 120 into 12 and 0, leaving the two strings sharing an even longer prefix.

The sequence is then pooled into one vector, and here the popular explanation is wrong. One changed token out of six does not move one sixth of the representation: token states are contextual, so one changed input perturbs every output state, and the pooling strategy varies by model anyway. The arithmetic of the average is not where the answer lives.

The training objective is. Contrastive training rewards the encoder for treating surface form as noise when meaning survives it, because it must: "how do I return a broken item" and "what is the process for sending back a defective product" have to land in the same neighbourhood or the model is worthless. That invariance is the entire product. What the objective never supplies is pressure in the other direction, the data or evaluation signal that would force this particular numeric difference to dominate, so it does not. On one catalogue corpus with one open model, the two chunks came out at cosine 0.98, while the gap between the correct chunk and a fluent, on-topic, completely wrong one was routinely under 0.02. When the distinction you need is smaller than the ambient spread of the space, ranking by it is a coin flip with extra steps. A model trained on component data might do far better. A general purpose one guarantees nothing, and that is what you design around.

Anisotropy compounds this, with a caveat. Transformer representation spaces tend to occupy a narrow cone rather than spreading over the sphere, which compresses cosine values into the top of the scale and makes raw thresholds hard to read. That finding came out of contextual language model representations though, and contrastive training pushes the geometry back toward uniform, so measure your own: sample a few thousand random pairs and plot the distribution. If unrelated pairs sit at 0.7, thresholding will not save you.¹²

None of which says a vector cannot encode an exact identifier. It plainly can. The practical problem is that general purpose encoders are not trained or evaluated to guarantee it, and there is a far cheaper component that gives you the guarantee outright.

What BM25 does instead

BM25 does not pool the sequence into one continuous representation. Indexed terms stay discrete symbols and each matching term contributes separately, as a function of how rare the term is in the corpus, how often it occurs in the document, and how long that document is. The frequency part saturates, which is what separates BM25 from naive TF-IDF, so a tenth occurrence buys much less than a second. (Learned sparse models do distribute weight across vocabulary dimensions. It is BM25 specifically that keeps terms discrete.)

Put numbers on it, on an electronics catalogue of 200,000 chunks. If you index the full designation as a single keyword term, sil 10-9 12k appears in exactly one chunk, so IDF is ln(1 + 199999.5/1.5), about 11.8. That is the number you want and it is available for the price of a field mapping.

Let the standard analyzer have it instead and you get fragments, which is less flattering but still works. 12k lands in maybe 180 chunks, IDF about 7.0. 120 lands in 6,400, because 120 is a popular number for ohms and nanofarads alike, IDF about 3.4. sil lands in 41,900, IDF about 1.6. The discriminating fragment still outweighs the packaging term four to one, with no training, no GPU and no tuning. It falls out of counting.³

The step everyone skips: getting the identifier out of the query

"Index a keyword field and you get the exact hit" is where most hybrid search writeups stop, and it is a hole rather than an answer. A keyword field is exact: it matches when the query term is byte for byte the stored value, after whatever normalizer you configured. Your user did not type the stored value. Your user typed:

Against a keyword field holding sil 10-9 12k, that matches nothing. Not approximately, not at a lower score. Nothing. The extra word is fatal, and so is a spacing or normalizer mismatch. Throwing the raw query at both fields does not extract the designation from the sentence around it. Something has to, and it will not be the search engine unless you tell it how. So pick a mechanism and own it:

• Extract, then boost. Run a catalogue lookup or a regex over the query, pull out anything shaped like a designation, and issue it as its own boosted term clause beside the normal analyzed query. This is my default: designations in a given catalogue almost always have a describable shape, and a lookup against the real product table beats any pattern.
• Emit both forms at index time. Configure the analyzer chain to produce the full designation and its components as terms in one field. More work up front, less query side logic to maintain.
• Dedicated identifier field with a normalizer, carefully. Apply the identical transformation to both sides, because a normalizer only helps if the query passes through it too. Do not reflexively strip separators: AB-12 and A-B12 collapse together, so do 1-23 and 12-3, and you have destroyed the distinction the field existed to protect. Keep the raw keyword field, add the normalized one beside it, and run a collision check across your whole designation inventory before indexing.
• Prefix or wildcard handling, deliberately. If people type three quarters of a designation, support that on purpose and pay for it on purpose. It is not free and it should not be an accident.

This is the most under-documented step in the entire pipeline. The sparse arm only does the work the dense arm cannot if the discriminating token reaches it as a term. Any code sketch that hides this inside a call named search(query) is hiding the hard part, which is most of them.

The analyzer will quietly undo all of this

None of this happens unless the term survives tokenization on the index side. The default standard analyzer in Elasticsearch and OpenSearch lowercases and splits on punctuation, so SIL 10-9 12k lands as sil, 10, 9, 12k. Two of those are bare integers matching tens of thousands of other products, and the one doing real work is the one nobody thought about. Version strings like 2.14.3 and designations like A-1 fare worse: after the split, nothing rare is left.

Run the analyze API against ten real identifiers from your corpus and read the terms it emits before you trust the sparse arm at all. It takes ten seconds. I have watched a team lose most of a week to the wrong hypothesis because nobody did, and the answer was that their document numbers were being split into four numeric fragments that each matched three thousand other documents.

The fix is a second field, and the two candidates are not interchangeable. A keyword field emits the whole value as one term, which is what gives you the exact designation. A whitespace analyzer splits at every space, so SIL 10-9 12k becomes three terms and never yields a full designation term at all, though it does preserve the punctuation inside each component that the standard analyzer destroys. So: keyword field for the complete canonical identifier, whitespace or custom analyzed beside it when partial matching earns its keep. A pattern capture filter can emit the full identifier alongside its parts, noting that token filters only see the stream the tokenizer produced, so the tokenizer has to hand it the material first.⁸⁹¹⁰¹¹

One tuning note nobody mentions

The default b = 0.75 penalizes long documents, and it comes from general document retrieval where length varies a great deal. Your chunks are all 400 tokens because you chunked them that way, so there is far less useful length variation to correct for, and what correction remains mostly punishes the chunks that packed in more content. On uniformly sized chunks, start at b between 0.3 and 0.5 and test from there. Leave k1 at 1.2 unless you have a reason.⁷

A reranker cannot recover what the chunker destroyed

Chunking has the highest ratio of impact to attention paid, and everything downstream inherits it. If the sentence that answers the question got separated from the identifier that makes it findable, no amount of clever scoring reunites them. Start with the mechanical failures, which are common and free to fix.

Count tokens, and count them with the right tokenizer

Splitting on 512 characters when your model counts 512 tokens gives you chunks a quarter of the intended size. Subtler: chunking with cl100k_base because the tutorial did, while your embedding model uses its own sentencepiece vocabulary, so 500 token chunks arrive as 560. What happens then depends on the wrapper: silent truncation, explicit truncation, or rejection. Silent is the common default and the worst case, quietly dropping the last two sentences of every chunk from your index. Prefer the setting that raises where one exists; where it does not, use the model's own tokenizer and assert on encoded length at ingest.

Split on structure, not on a window

Recursive splitting that respects document hierarchy, headings then paragraphs then sentences then a hard character cut as last resort, has beaten a fixed sliding window on every corpus I have tried. Fixed windows cut mid sentence and mid table, which is where the damage happens.

Tables are the specific killer here. A product table gets sliced between rows, the part number lands at the bottom of chunk 14 and its attributes, the ones that answer the question, at the top of chunk 15. Both chunks are individually plausible and neither is useful. Detect table structure at ingest, keep rows intact, and repeat the header row into each chunk. The header costs you tokens and buys you a chunk that stands alone.

Sizes I actually use

The overlap column earns a second of thought. A rare term landing in an overlapping region now appears in two chunks instead of one and usually gives up some of the IDF advantage that made it useful. Usually, not mechanically: the formula depends on N as well as n(t), and overlap raises the total chunk count too, so the size and occasionally the direction of the effect depend on how the collection shifts overall. Noise on a prose corpus. Measure it on the identifier slice of anything built around rare designations.

Metadata is retrieval, not decoration

Attach document id, section path, effective date and any entity ids you can extract cheaply, then filter before you score. A query about the current policy should not be ranking against three superseded versions and hoping the reranker sorts out which one is live. Half the queries that look like hard semantic problems are date filters wearing a costume.

Do not add raw BM25 and cosine scores

This is where most hybrid implementations go wrong, for a boring reason: the two arms produce numbers on different scales and different kinds of scale. Cosine is bounded in [-1, 1] and in practice compressed into 0.6 to 0.95. BM25 is unbounded, depends on corpus statistics and query length, and routinely runs 4 to 40 inside one result set. Adding them is arithmetic on incompatible units.

Start with reciprocal rank fusion. Reach for score normalization only when you have enough labelled data to calibrate it, which for most teams means later and for some means never.

Normalizing is harder than it looks. Min max per query is defined by the extremes of the result set, so one outlier at BM25 60 rescales everything beneath it and your ordering starts depending on a document that never reaches the top 10. Z score behaves better and still inherits noisy estimates from small candidate sets, mismatched distribution shapes between the arms, and the question of what a document missing from one arm scores. If you do calibrate a convex combination the form is α·norm(dense) + (1−α)·norm(sparse): prose corpora put me between 0.6 and 0.75, identifier heavy corpora have taken me to 0.4 and once to 0.3.

What RRF throws away

RRF sidesteps the scale problem entirely by discarding scores and using only positions.

⁵ Cormack, Clarke and Buettcher introduced this and the convention of a constant near 60.

At the conventional k = 60, the gap between rank 1 and rank 2 is 1/61 minus 1/62, about 0.00026, while rank 1 to rank 20 is about 0.0039. RRF is deliberately flat at the top: it says "both arms liked this" far louder than "one arm loved this." Usually what you want, occasionally exactly wrong, because when BM25 lands an exact identifier at rank 1 with an IDF of 11.8 and noise beneath it, that magnitude was real information and RRF has discarded it.

Two levers. Lower k, which sharpens the top far more than it looks: at k = 20 the rank 1 to rank 2 gap is 1/21 minus 1/22, about 0.00216, roughly 8.2 times the gap at k = 60, not the two or three you would guess from the formula's shape. Or weight the arms, w_sparse = 1.4 against w_dense = 1.0 on an identifier heavy corpus. Both come off measurement, never off a paper: sweep in steps of 0.1, plot coverage at your fusion depth, prefer a plateau to a peak.

Candidate depth, and why it is not one number

Depth before fusion sets a hard ceiling on everything downstream, because fusion cannot promote a chunk neither arm returned and neither can the reranker. So "how deep" is really "what coverage do I need at the top of the funnel," which is measurable rather than guessable.

"Take 100 from each arm" is the number that circulates, and it is a heuristic wearing a rule's clothing. Past the point where an arm stops surfacing anything relevant, extra candidates add retrieval and fusion work and change which chunks survive into the pool, which is the part that can quietly hurt you. What they do not do is make the reranker more expensive: the cross encoder scores whatever fuse_to is, 50 in the sketch below, whether the arms handed fusion 160 candidates or 400. Reranker cost tracks the post-fusion cutoff, not retrieval depth.

Nor is the sparse arm a narrow gadget that only fires on exact designations, a framing that undersells it badly. BM25 earns its place on named entities, rare technical vocabulary, verbatim error messages, quoted phrases, and any query whose wording tracks the source closely, which in internal documentation is a great many queries. BEIR found it a stubbornly strong zero-shot baseline across heterogeneous tasks for exactly this reason.⁶

So tune the arms separately: plot coverage against depth for each one, and take the knee rather than a round number. On the catalogue corpus behind FIG.02 the sparse curve flattened around 40 while the dense one kept paying out past 100. On a support ticket corpus the shape came out closer to the reverse.

Deduplication happens during fusion, not after it. You accumulate into a map keyed by chunk id, so a chunk arriving from both arms has its contributions summed, and that summing is the entire point. Agreement between two independent retrieval strategies is real evidence. Take the max instead and you throw that evidence away, then wonder why fusion is not doing much.

# two arms, per-arm depth, weighted RRF, then a cross encoder.
# every number below is one corpus's answer, not a default. yours will differ.

def sparse_query(query):
    # the step most sketches hide. a keyword field is EXACT: it will not
    # find "sil 10-9 12k" inside "SIL 10-9 12k pinout" on its own.
    ids   = extract_designations(query)      # catalogue lookup, regex fallback
    parts = [{"match": {"text": query}}]
    for d in ids:
        parts.append({"term": {"designation": {"value": normalize(d), "boost": 8}}})
    return {"bool": {"should": parts}}

def hybrid(query, k_rrf=60, w_sparse=1.4, w_dense=1.0,
           depth_sparse=40, depth_dense=120, fuse_to=50):
    # depths are separate on purpose: the sparse curve went flat early,
    # the dense one kept paying out. both were measured, not assumed.
    sparse = bm25.search(sparse_query(query), size=depth_sparse)
    dense  = ann.search(embed(query), k=depth_dense)     # similarity per model card

    # dedup happens HERE, during fusion: same chunk from both arms sums.
    # agreement between arms is the signal. do not take the max.
    scores = defaultdict(float)
    for arm, w in ((sparse, w_sparse), (dense, w_dense)):
        for rank, hit in enumerate(arm, start=1):
            scores[hit.chunk_id] += w / (k_rrf + rank)

    fused = sorted(scores.items(), key=lambda x: -x[1])[:fuse_to]
    return fused

def rerank(query, fused, n=8):
    pairs = [(query, chunk_text(cid)) for cid, _ in fused]
    # 50 pairs at batch_size=32 is two batched passes, not 50 calls.
    # the work scales with pairs; the latency scales with how you batch them.
    scores = cross_encoder.predict(pairs, batch_size=32)
    order  = np.argsort(-scores)[:n]
    log_candidates(query, fused, scores, order)   # log both orderings. always.
    return [fused[i] for i in order]

Sketch, not a library. The parts that matter are the depth, the weights, and the log call.

What a cross encoder does, and what the pricing page implies it does

The architectural difference is the entire story. A bi-encoder, which is what your retrieval embedding model is, encodes query and document separately and compares the resulting vectors. The document side never sees the query, so every document vector can be precomputed at ingest and searched with an ANN index in milliseconds. That independence is what makes retrieval fast and also what makes it blunt: the model committed to a representation of the document before it knew what anyone would ask.

A cross encoder concatenates query and document into one sequence and runs full attention across the pair, so every query token attends to every document token. It can notice that the numbers differ in the last character, because they sit in the same attention context. That is why it ranks better, and why it costs what it costs: nothing precomputes, and you pay one scored pair per candidate, at query time, on every query.

The latency math

Fifty candidates is fifty pairwise evaluations, and the work scales with those pairs, but that is not fifty forward passes: at batch size 32 the runtime does two batched passes, one of 32 pairs and one of 18. Worth the precision, because batching is where this is won or lost. On one setup, a small bge-reranker-base class model at batch size 32 on a mid range GPU landed in the 40 to 120 ms range, with the hosted API equivalent around 100 to 300 ms including round trip. One datapoint, not a spec sheet: precision, batch size, sequence lengths, model version, runtime and card all move it, several by more than a factor of two.

Whether that cost matters is a question about your product, not about reranking. Above it disappears next to two seconds of generation. In a search product with a 200 ms budget, no streaming and short answers, the same 120 ms is most of your budget and the reranker becomes the thing you optimize.

What is not acceptable anywhere is sending candidates one at a time, getting 1.5 seconds, and concluding reranking is too slow for production. Batch them. It is the most common self inflicted performance wound in reranking code and it looks exactly like a model problem until you check the call pattern.¹²

The truncation trap

Many BERT family cross encoders take 512 tokens for query and document combined, and that number circulates as though it were a property of reranking rather than of one model generation. It is not. Current hosted rerankers span from roughly a thousand tokens on some traditional pointwise models to tens of thousands or more on long context and listwise architectures, and the spread inside a single vendor's lineup is now as large as the spread between vendors. Look up two numbers for the specific model you are calling: what the model can represent, and what the hosted endpoint actually permits.¹³¹⁵

How you find out you exceeded it varies, which matters more than it sounds. Some hosted APIs expose a truncation switch you can disable so overlength input raises instead, and Voyage documents exactly that. Local libraries generally do not: Sentence Transformers truncates to the configured maximum with no equivalent switch, so build it yourself by tokenizing the pair and asserting on the combined length before inference. Ten lines, and a slow mystery becomes a stack trace.¹²¹³

Either way, check it against chunk size. If you grew chunks to give the generator more context, verify in the same commit that you did not blind the reranker. I have watched a chunking change improve generation quality and degrade ranking quality simultaneously, which cost a confusing week of A/B results before anyone checked a truncation limit.

The part I am tired of

Here is the sales pattern that has worn me down. A reranker vendor shows an nDCG@10 improvement on BEIR, a collection of public datasets that essentially every retrieval model of the last several years has been developed against, then lets you infer the same delta on your internal wiki. Do not assume it transfers. On a corpus the model has never seen, in a domain whose vocabulary it does not know, against queries written by users rather than annotators, I have measured the same reranker delivering about a third of the headline gain, and on another project nothing distinguishable from noise.

The uncomfortable part is that the vendor is not lying. The number is real and reproducible. It is a number about BEIR, and BEIR is not your company. The honest slide would say "here is the gain on public benchmarks, here is how to measure it on yours, and here are the corpus characteristics where we would expect it to be small." Nobody ships that slide.

The billing unit is not standardized either. As of August 2026 Cohere prices rerank by search, Voyage by processed tokens with the query re-charged for every document, Pinecone by request, others in credits. Check yours before you model the cost, and note the direction they share: longer candidate lists cost more, and the documentation recommending a generous candidate list is written by the party that benefits from one. Nobody in that transaction is incentivized to tell you your coverage at 50 is already 0.94 and you could be sending 25.¹⁴¹⁶¹⁷

And the claim that gets muddled most: a reranker cannot improve coverage of the pool it was handed, but it very much improves recall at your final cut, which is the entire point. Moving a chunk from rank 34 into the top 8 raises Recall@8 without changing Recall@50 by a single document. So say which k you mean. "Rerankers do not improve recall" is true at one depth and false at another, and the ambiguous version gets used to justify both buying and not buying the same product.

Aside / log both orderings or you are debugging blind

The insurer bot from Part 1, the one that cited a policy exclusion that did not exist, stayed unfixable for a dull reason: nothing was logged, so nobody could say which chunk the claim came from. Reranking makes that worse, because now there are two orderings and the trace shows you the second one.

Log the fused list with both arm scores, log the post-rerank list with the cross encoder scores, keep chunk ids in both. Forty lines. Every hour I have spent on this has paid for itself within a month, which I cannot say about most of the tooling I have bought.

How I learned this

As I admitted in Part 1, I did not arrive at hybrid retrieval through insight. I told a client their keyword index was legacy infrastructure that semantic search subsumed, and it cost them about two weeks when the warehouse staff who actually used the bot turned out to type part numbers into it. What I got wrong was not "semantic search is bad." It was assuming that a model which handles meaning must therefore handle an exact string, when the training objective that produces the first is the one that dissolves the second. Every piece of that explanation was available to me. The demo worked, so I stopped looking.

One variable, a frozen eval set, and write it down

The fifty queries from the opening section only take you so far. They are excellent for finding what is broken and insufficient for sweeping weights, choosing a reranker and reporting an improvement, because after the third sweep they are a development set. You have fitted to them, and any number you quote off them is a training number in an evaluation number's costume. Three sets:

• The diagnostic set. 50 real failures. Read by hand. Its job is to generate hypotheses, and it can be as biased as it likes, because you are not reporting numbers off it.
• A development set. A few hundred queries with judged relevance, sampled to look like traffic rather than like failures. This is what you sweep against, repeatedly, and it is expected to get overfitted.
• A held out test set. Touched rarely, ideally at release boundaries. Bootstrap a confidence interval over it, because a two point difference on 200 queries is very often nothing at all.

"Pick the plateau, not the peak" is a useful instinct that does not solve this. It reduces how hard you fit to a small set. It does not stop the set being reused. Then change one thing at a time, in this order, because each stage inherits the ones before it:

1. Chunking and ingest. Measure per-arm coverage at your retrieval depth.
2. Sparse analyzer and field mapping. For a change confined to the identifier field, report the identifier slice separately, since that is where the effect should concentrate. For a change to the general analyzer, check every category it can touch: quoted phrases, technical vocabulary, punctuation handling and ordinary lexical queries all move when token boundaries move.
3. Fusion weights and k. Measure pooled coverage at the fusion cut.
4. Reranker choice and final cut. Measure at the depth you actually ship: if the generator gets 8 chunks, that is nDCG@8 and Recall@8, not nDCG@10 borrowed from a leaderboard. Then measure answer correctness, which is the only one of the three the user experiences.

Order matters because tuning fusion weights on top of bad chunks gives you the weights that are optimal for bad chunks, and fixing chunking afterwards means redoing the sweep. Ask me how I know.

Aside / still no golden dataset

Part 1 admitted I had been meaning to build a golden dataset for my own side project for eight months. It is now nine, and I have just described a three set protocol I am running zero sets of. The collapse always happens the same way: the diagnostic set exists because something broke, then it is the only labelled data in the repo, so it becomes the dev set by default and the test set by neglect. Nobody decides to do this. You just never build the second one.