I benchmarked my BPE tokenizer against tiktoken and BERT: the numbers were not kind
I retrained the BPE from transformer-from-scratch to an 8,000-token vocabulary and pitted it against cl100k_base, o200k_base, GPT-2 and multilingual BERT on the same English, Spanish and code corpora. The Spanish gap was worse than expected, and the real reason was not the one I had assumed.
When I wrote my own BPE tokenizer for the from-scratch transformer, I learned how it works from the inside: count pairs, merge the most frequent one, repeat. What I did not know was how much worse that algorithm — mine, unchanged — behaves outside the language it was trained on. Instead of assuming, I built a reproducible benchmark that measures it with real numbers, against four production tokenizers, on the same texts.
The five contenders. My BPE, plus cl100k_base and o200k_base from tiktoken (the tokenizers behind GPT-3.5/4 and GPT-4o), plus GPT-2 and bert-base-multilingual-cased via Hugging Face's tokenizers library. The checkpoint I trained in transformer-from-scratch has a 512-token vocabulary, and comparing that against vocabularies of a hundred thousand or more would only measure that 512 is a lot smaller than a hundred thousand, which is not information. So I retrained the same algorithm, without touching a line, on the same tiny-shakespeare corpus, up to an 8,000-token vocabulary (7,744 learned merges): still an order of magnitude smaller than the rest, but large enough that the comparison is now about what language it was trained on, not just its size.
Corpora with real substance. Three public-domain texts, each trimmed to a fixed, documented size: Pride and Prejudice for English, Don Quijote for Spanish (both from Project Gutenberg, with license headers stripped), and a code corpus built by concatenating sorted modules from Python's standard library. Every download is pinned to a sha256 checksum in the script itself, so the benchmark reproduces exactly a year from now. All three corpora end up almost the same size, around 700KB each, so text length is not the variable deciding anything.
What wasn't in the plan: 56 seconds for a single pass. The first time I ran the full benchmark I let it go with a five-minute cap, and the five minutes ran out without a single line printed. I measured the problem directly: my encode() rescans the whole token sequence on every merge it applies, so its cost grows faster than the text length. A single pass over 150,000 bytes of English — its best-matching language — took 56 seconds. Doing that six times (one warmup plus five timed runs, the speed methodology) over the three full 700KB corpora would have taken, for that one tokenizer alone, well over half an hour. The fix was measuring all four metrics on a fixed 100,000-byte sample per corpus, the same sample for all five tokenizers, documented in the README rather than hidden. The whole benchmark went from never finishing to taking seven minutes, start to finish.
Fertility, with the real numbers. My BPE needs 2.12 tokens per word in English, 2.83 in Spanish and 5.97 in code: a 33.6% jump in Spanish and a 181.4% jump in code relative to its own English, the worst degradation of the five. cl100k_base goes from 1.40 to 1.73 (+23.6%) and to 2.29 in code (+63.4%). o200k_base, the newest, goes from 1.39 to 1.54 (+11.4%) and to 2.30 in code (+66.3%). The numbers back up the usual claim about modern, multilingual tokenizers — that they suffer far less outside English — but they also show that "far less" is not "none": not even o200k_base is neutral across languages.
Vocabulary size does not tell the whole story. bert-base-multilingual-cased has 119,547 tokens, under 60% of o200k_base's 200,019, and yet its English-Spanish gap is the smallest of the five: +6.9% against o200k_base's +11.4%. What explains it is not size, it is where the data came from: Wikipedia in 104 languages, with Spanish present in a share close to English's, not a leftover fraction. A bigger vocabulary buys more distinct tokens; which language they get spent on is decided by the training corpus, not the entry count.
A surprise I almost published without checking. bert-base-multilingual-cased — the only one of the five with a real [UNK] token — has a higher unknown-token rate on English (3.585%) than on Spanish (0.456%) or code (0.015%), exactly backwards from what you would expect. Before writing "BERT struggles more with English", I pulled the exact spans that were producing [UNK]: all 842 occurrences in English are curly quotation marks (“ ” ‘ ’), because that particular Gutenberg edition of Pride and Prejudice uses them and a 2018 WordPiece vocabulary does not include them. All 124 in Spanish are em dashes (—), which the Don Quijote edition uses to mark dialogue. It is not a language-coverage problem at all, it is the typography of that specific edition — and skipping the check would have shipped a false conclusion dressed up as data.
Another crack between two metrics that should tell the same story. By fertility, code is the worst case for all five tokenizers, but that partly measures a flaw in the metric itself: splitting on whitespace turns self.assertEqual(x, y) into a single "word" that then explodes into a dozen tokens. By bytes per token, the picture flips: cl100k_base and o200k_base — the only two trained with real code in the mix — compress the Python corpus more than English prose (4.55 versus 4.37 bytes/token, and 4.52 versus 4.42). The other three, mine included, compress code worse than English. Fertility and bytes per token do not always agree, and when they don't, it is usually because one of them is measuring something else.
All of it — the code, the checksum-pinned corpora, the full fifteen-combination result set and the scripts to reproduce it from zero — lives in the repository. I did not run this benchmark to prove my tokenizer is worse; I already knew that in the abstract. I ran it to get concrete numbers for how much worse, and to learn that the right answer is almost never "the vocabulary is small": it is the language you learned to merge bytes in.
