BM25 versus embeddings: I added semantic search to my search (with real numbers)

A while ago I wrote about how this site's search works with BM25: counting words with judgement, no model involved. I closed that article with an open question — what does the next step up actually buy, when vectors replace counted words — and promised the lab would answer it. This time I answer it inside the search itself: there is a new toggle, off by default, called "Semantic", and this is what it does, what it costs, and when it actually helps.

The idea is to re-rank, not replace. BM25 is still the first filter: for every question it computes its usual best-scoring 50 fragments out of the 876 the index currently holds, exactly as before. What's new happens only if you flip the toggle: the question becomes a 384-number vector with MiniLM (Xenova/all-MiniLM-L6-v2, the same model the lab's embeddings map and sentence comparison demo already load), compared by cosine similarity against a precomputed vector for each of those 50 fragments, and the final score is half normalized BM25, half cosine. Dedup by page and the usual language boost still apply, unchanged.

The vectors travel precomputed and quantized. A new script, `build-ask-embeddings.mjs`, runs the same model in Node over the index's 876 fragments and writes `ask-embeddings.bin`: each vector is quantized to 8-bit integers with a single global scale (127 divided by the highest absolute value across the whole corpus, with a hair of headroom) documented in a JSON header at the start of the file itself. In float32 those vectors would weigh 1.3 MB; quantized, they weigh 329 KB. The browser downloads them once, dequantizes by dividing by that same scale, and from there they're ordinary numbers.

Order matters more than ids here. The embeddings file repeats no URL and no id: the vector at position 407 is simply the embedding of fragment 407 in `ask-index.json`. That's more compact, but it means the two files have to be published together, in the same order — so the publish flow is now `build-ask-index.mjs` followed by `build-ask-embeddings.mjs`, in that order, and the search checks on every load that the vector count matches the fragment count. If the two ever get generated separately and drift out of sync, the search notices, warns in the console, and keeps working on BM25 alone rather than re-ranking with vectors that no longer line up with the right fragments.

Before turning it on by default, I measured it. I wrote 15 real questions, about half using a different word than the content itself uses, a couple mixing English ML jargon into a Spanish sentence, and a couple of controls where BM25 should already win unaided. For each one I compared BM25's own top-3 against the hybrid's top-3 and judged by hand which set actually answered the question. The full table — all 15 questions, both top-3s, and the reasoning for each verdict — lives in the repository; here are the numbers that actually matter: 1 clear hybrid win, 2 small wins, 8 ties where BM25 already sufficed, 1 hybrid loss, and 3 questions where neither approach found what it was looking for.

The clear win happened in English, not Spanish. "How do I know if my customers are about to leave" shares not one word with the project that should answer it (`customer-churn-prediction`), and yet MiniLM's cosine ranks it as the single best match across all 876 chunks in the index. BM25 had it at position 9; the hybrid promoted it to position 2. But the exact same question in Spanish — "cómo sé qué clientes van a dejar de usar mi servicio" — didn't get that happy ending: neither BM25 nor the hybrid put it in the top-3. The lab's own sentence-comparison demo already warns about this with its own examples: this model is trained mostly on English, and here it shows up in a real case, with a whole site behind it.

For genuinely conceptual leaps, MiniLM doesn't make them either. I tried "how do I stop my model from memorizing the data instead of generalizing", which should point at the article on data leakage between train and test. No surprise there by keyword count: BM25 ranks it 8th out of 161 candidates. What I didn't expect is that pure cosine — comparing the question against all 876 fragments, with no help from BM25 — ranks it 22nd, worse than plain keyword counting. With "is my new model actually better or just lucky", which should surface real regression or noise in your eval, pure cosine sends it to rank 82 of 876. And with a question about a chatbot that shouldn't invent things, the word "chatbot" pulls the vector toward the transformers article instead of the verifiable-RAG one, rank 107 of 876. No blending trick fixes this: if the question's own vector already points away from the right fragment, adding half of a weak BM25 score doesn't close that gap.

A subtler failure: it scores the fragment, not the page. A long page has several fragments, and BM25 and cosine can each prefer a different one of them — dedup by URL happens after scoring, so the fragment that competes is the one BM25 already picked, not the best fragment either metric could offer for that page. I saw this with concrete numbers: for the Spanish customer question, the `customer-churn-prediction` fragment that BM25 brought into the candidate pool scored only 0.351 cosine against that question, even though a different fragment from the same page reaches 0.572. With the 50/50 blend that comes out to a hybrid score of 0.539, well below the 0.813 and 0.776 its two competitors scored. A redesign that scored per page — the best cosine across all of a page's fragments, not just the one BM25 already chose — would likely fix part of this. I didn't build that this time: I wanted to measure what I'm actually shipping, not what I could ship with more time.

The 50/50 blend won by elimination, not intuition. I also tried 65% BM25 / 35% cosine and 35% BM25 / 65% cosine on the same 15 questions. The BM25-heavy version barely moved anything — the English customer question stayed at rank 3 instead of 1 or 2. The cosine-heavy version did rescue those cases better, but at the cost of breaking controls where BM25 already got it right: on "is my model better or just luck" it pulled in an unrelated transformers article, purely from vague shared vocabulary about "model" and "training". The even split was the only one of the three that never made a control case worse while still delivering the clear rescue.

Which is why the toggle stays off by default. Out of 15 real questions, BM25 already sufficed unaided in 8 of them — most questions about this site share vocabulary with their answer, because they're about specific named things (evalgate, tokenmeter, BM25) that appear the same way in the question and in the text. Semantic search costs a ~20-25 MB model download the first time — the interface says so plainly before you turn anything on — and it only earns that cost when a question uses different words than the content does, especially in English. If you'd rather see it running: the question-mark button in the bottom right has the "Semantic" toggle in its header; turn it on, ask something in your own words, and compare.