Everything that happens when you press Enter in my search
The technical version: from the key event to the ranking, with the 735-fragment index, BM25 with k1=1.5 and b=0.75, the language boost and the decisions made along the way. For readers of the jargon-free version who want to see the gears.
A few days ago I explained without jargon how this site's search works. This is the other half: the version with the gears exposed, number by number, for anyone who wants to build their own.
The index is rebuilt on every publish. A script walks projects, tools, articles and pages, and chops everything into paragraph-sized fragments. Right now that yields 735 fragments covering about a hundred pages in two languages; each stores its text, URL, title and language. It all travels as a single static JSON that the CDN compresses — the browser fetches it once, the first time you open the search, and it stays cached.
Tokenisation decides what counts as a word. Before anything is compared, question and fragments go through the same mill: lowercase, accents stripped (so "qué" and "que" match), and stopwords removed for each language — articles, prepositions and other filler that appears everywhere and distinguishes nothing. The stopword list is double, Spanish and English, because the index mixes both.
The ranking is BM25, with its two stock dials. For every term in your question, each fragment accumulates a score combining three things: how rare the term is across the index (rare terms rule), how often it appears in the fragment (with saturation: the fifth occurrence is worth far less than the first — that is k1=1.5), and how long the fragment is (long ones pile up occurrences by sheer length, which b=0.75 penalises). I used the classic literature values and left them alone: with an index this small, tuning them moved nothing.
One decision of my own: the language boost. Ask in Spanish and Spanish fragments multiply their score by 1.35. That is not theory: without it, a Spanish question about a project sometimes returned the English version of the same page — correct but awkward. The factor came from trying: 1.2 did not break the ties and 1.5 buried legitimately better English results.
The finishing touch: dedupe by URL. Several fragments of the same page can crowd the top. Before showing anything, only the best fragment per URL survives, so you get four different pages instead of four pieces of the same one. It is one line of code and it is the difference between a search that helps and one that repeats itself.
What it actually costs. The whole cycle — tokenising the question, scoring 735 fragments, sorting and deduplicating — runs on the main thread in well under a frame; the only real cost is the first download of the index. No server, no API, no per-query cost, and your question never leaves the browser. For a site this size, anything fancier would solve a problem I do not have — and in the lab you can see exactly what the next step up buys, when vectors replace counted words.
