How this site's search finds me (with no servers)
Bottom right there is a button that answers questions about this portfolio. It calls no server: it scores 600 text fragments in your browser with a formula from the nineties that still wins battles. Here is how it works, jargon-free.
In the corner of this site lives a button with a question mark. You ask it "what is evalgate?" and it hands back the exact three or four paragraphs of the site that answer it, each with its link. The interesting part is not that it works: it is that no server is thinking up the answer. Everything happens inside your browser, and the trick fits in a short article.
First, the site gets chopped up. Every time I publish, a script walks all the content — projects, tools, articles, FAQs — and splits it into about 600 paragraph-sized fragments, each tagged with the page it came from. That file travels to your browser when you open the search. It is literally the whole site, ready to be queried.
Then, counting words with judgement. When you type a question, the search counts how many times each of your words appears in each fragment. But raw counting misleads, so it applies two corrections with a formula name (BM25) and common-sense logic: rare words are worth more than common ones — "evalgate" says far more than "project" —, and repeating a word twenty times in a fragment does not make it twenty times better, because relevance saturates.
That is all. Really. Every fragment gets a score, they get sorted, and I show you the best ones with their links. No language understanding, no magic: well-considered arithmetic over counted words. This formula was published in the nineties and is still the backbone of a great many serious search engines, because for finding text that contains your words, very little beats it.
Why not something smarter? I could use a model that understands meaning, like the one in the lab's word map — in fact that is what I do in my document assistant. But the honest question here was different: what is the simplest thing that solves THIS problem well? For searching a 90-page site, the answer is this formula: instant, free, and your question never leaves your device. Choosing the small tool when it suffices is the engineering lesson I am finding hardest to learn.
Next time a search engine finds you what you were looking for, you know what sits underneath half the time: words counted with judgement. And if you want to see mine in action, the question mark is still down there on the right.
