Late Interaction, SPLADE, and the Architectures I Still Do Not Ship
Addendum to Part 2. The four families between a bi-encoder and a cross encoder, what each one would have to prove, and the number that keeps deciding it for me.
Part 2 argued for a boring pipeline: BM25 and dense retrieval, fused deliberately, then a cross encoder once the measurements say ordering is the failing stage. It also drew that pipeline as two boxes, which is the standard picture and is not the whole design space.
I cut this material from Part 2 because it answered a different question. The main article explains what I recommend. This one examines the architectures that could eventually make me change that recommendation, and why they have not done so yet.
Four families that live between the two boxes
A bi-encoder encodes query and document separately, so document vectors precompute at ingest and search in milliseconds, and the model had to commit to a representation before it knew the question. A cross encoder concatenates the pair and runs attention across both, which ranks better and costs one scored pair per candidate at query time. Those are the two ends. Between them:
- Late interaction. ColBERT and its descendants precompute per token document representations at ingest, then do fine grained query to token matching at search time. This is a third position rather than a compromise: you keep most of the interaction and most of the precomputation.1
- Multi-vector retrieval. Several vectors per document, scored with a max over them. Cheaper than per token, more expressive than one pooled vector.
- Poly-encoders. The query context is represented by several learned codes and one vector is cached per candidate; the candidate vector then attends over those context codes before scoring.2
- Learned sparse. SPLADE and relatives come at it from the other end entirely, keeping the inverted index and learning term expansion into it.3
Why I still ship two boxes
Cost and operational familiarity, not a finding that the middle ground is worse. Late interaction indexes are substantially larger, and that has been enough to send me back to BM25 plus dense every time I have looked. Which makes it a budget conclusion rather than a technical one, and those are easy to confuse from the inside.
It is also convenient that my measurements keep endorsing the architecture I already know how to operate. That is the kind of coincidence I would interrogate in someone else's writeup, so I should say plainly that I have never separated "this performed worse" from "this was unfamiliar and the bill was larger." Those feel identical from the inside and they are not the same finding.
SPLADE is not a guaranteed exact match mechanism
This one matters because of what Part 2 was about. If learned sparse gave you exact identifier matching plus semantic expansion out of a single index, the two arm architecture would be a workaround for a limitation that no longer exists.
It does not, quite. The original and common BERT based SPLADE models expand into a fixed WordPiece vocabulary, and the published weaknesses cluster around vocabulary gaps and low frequency terms, which is precisely where an unusual product designation lives. Learned sparse as a broader category is not inherently confined to that vocabulary and expanded vocabulary variants exist, but coverage and rare term behaviour stay on the list of things to test rather than assume.4
So if your corpus is full of designations, do not swap out the sparse arm for a learned sparse model on the assumption that exact matching comes along for the ride. Measure it on your own identifiers first.
Aside / the leaderboard has a disclosure problemMTEB has been public and influential long enough that I no longer treat every leaderboard score as equally zero-shot. Some models disclose training on benchmark training splits. For others the training mixture is not public enough to rule out overlap, and MTEB's maintainers now track a zero-shot measure for exactly that reason.6
That does not make the leaderboard useless. It changes what the number proves. A high score shows a model performs well on MTEB under the reported evaluation. It does not by itself show the model will generalize to your private corpus, or even that every model in the table reached its score with the same degree of exposure to the benchmark's distributions.
So my standard is simple: use MTEB to pick candidates, not winners. The winner gets decided on a held-out corpus the model developer could not have trained against.
How much weight do you give a leaderboard result when the training mixture is undisclosed? And has anyone actually measured whether leaderboard order survives on a genuinely private corpus?
Contextual prefixes, and reading the published numbers correctly
The chunking technique that has moved my numbers most is prepending 50 to 100 tokens of document level context to each chunk before embedding it, generated once at ingest with a cheap model: "this chunk is from the 2025 returns policy, section 4, covering damaged goods." The chunk that read "this must be reported within 14 days" becomes retrievable by someone asking about damaged goods, which it was not, because the pronoun chain that gave it meaning lived two paragraphs up.
Be careful with the published numbers. Anthropic's contextual retrieval writeup reports about a 49% reduction in failed retrievals, but that figure is contextual embeddings and contextual BM25 together, and the larger figure adds reranking on top. Take one piece of the recipe, expect one piece of the result.5
My own gains have been smaller and concentrated almost entirely in documents heavy on pronouns and cross references. On a flat product catalogue it did close to nothing, because nothing was missing: each row already stood alone. It costs real money at ingest and makes reindexing more expensive, so it is a decision, not a default.
What would change my mind
A run of late interaction or learned sparse against a properly tuned hybrid baseline, on a corpus that is not MS MARCO or BEIR, with the index size and ingest cost reported alongside the retrieval numbers rather than omitted from the blog post. That last part is where every comparison I have read goes quiet, and it is the number that decided it for me.
If you have run that, I want to hear what happened, including and especially if the answer was "it worked and it was not worth the index."
References
- Khattab, O., and Zaharia, M. (2020). ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. SIGIR 2020. ↩
- Humeau, S., Shuster, K., Lachaux, M.-A., and Weston, J. (2020). Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. ICLR 2020. ↩
- Formal, T., Piwowarski, B., and Clinchant, S. (2021). SPLADE: Sparse lexical and expansion model for first stage ranking. SIGIR 2021. See also Formal, T., Lassance, C., Piwowarski, B., and Clinchant, S. (2021), SPLADE v2, for the later variants and their BEIR results. ↩
- Iida, H., and Okazaki, N. (2022). Unsupervised domain adaptation for sparse retrieval by filling vocabulary and word frequency gaps. AACL-IJCNLP 2022, 752–765. doi:10.18653/v1/2022.aacl-main.57. The source for the rare term and vocabulary gap problem. ↩
- Anthropic (2024). Introducing contextual retrieval. Anthropic Engineering, 19 September 2024. Source of the 50 to 100 token prefixes, the 49% figure for contextual embeddings plus contextual BM25, and the 67% figure once reranking is added. ↩
- Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. (2023). MTEB: Massive text embedding benchmark. EACL 2023, 2014–2037. doi:10.18653/v1/2023.eacl-main.148. Cited for the benchmark itself and for its finding that no single method dominates every task. For the zero-shot annotation and the disclosure question, see the current leaderboard rather than the paper: the measure was added after publication and the per-model values change as submissions do. ↩