A multilingual tokenizer cuts token counts—but model benefits remain untested

A sentence that looks equally long on screen can take very different numbers of tokens to process, depending on the language and tokenizer. That matters because language-model systems often measure context limits and usage charges in tokens. If one language takes more tokens to express the same content, users may fit less text into a context window or pay more for a request. But reducing token counts is only one step: it does not, by itself, show that a model will understand or generate that language well.

In an arXiv preprint, Johin Johny, Ankur Ghosh and Shervin Sam of Havon AI describe tokenizers designed for many languages. Their central question was whether one could reduce token use across languages—including English—while maintaining favorable token counts for Indian languages.

The researchers trained their tokenizers from scratch using human-written web, educational and programming text. Havon v3 uses byte-pair encoding, a method that learns which character sequences to combine into reusable pieces. Its text-processing rules keep certain Indic vowel marks attached to their letters. For languages usually written without spaces, such as Khmer and Burmese, it divides text into bounded chunks rather than treating a long uninterrupted passage as one unit.

Havon v4 adds a second training stage based on SuperBPE. Unlike the usual approach, which prevents a token from crossing a space, this stage can combine multiple neighboring words. The team compared token counts with GPT-4o’s o200k_base tokenizer on FLORES-200, a multilingual translation benchmark. They counted tokens for individual sentences in the devtest split, which contains 1,012 sentences per language. A lower count means fewer tokens for those benchmark sentences; it does not measure translation quality.

At the same vocabulary size as GPT-4o—200,019 tokens—Havon v4 used fewer tokens on English and on 161 of the 204 languages tested. Its median count across the 25 Indian languages was 0.74 times GPT-4o’s count, or about a quarter fewer. The authors report lower counts for all 25. By contrast, v3 at the same vocabulary size used more tokens on English and had lower counts on 133 languages. This comparison points to a benefit from the multi-word stage. In a separate controlled test on identical text, adding that stage also reduced counts in the six languages examined.

With a larger vocabulary of 262,144 tokens, v4 used fewer tokens than GPT-4o on 182 languages. At 524,288 tokens, it did so on 203 of 204; German was the exception. Havon v5 adapts the recipe using programming code in 15 languages. On the code samples tested, its counts were closer to GPT-4o’s than v4’s, though they remained higher in the reported categories. V5 also had lower counts on all 25 Indian languages and used 0.94 times GPT-4o’s tokens on English.

The results varied by language. The authors report especially strong reductions for Indian languages, English, Hindi and Romance languages. Languages written without spaces tended to gain less from combining words across spaces; at equal vocabulary size, v4 sometimes needed more tokens than v3 for languages including Khmer, Burmese and Tibetan. A fixed-size vocabulary is a limited budget: adding useful pieces for one language can leave fewer available for another. The authors’ reallocation experiment illustrates this trade-off, improving counts in some languages while worsening them in others.

Several limitations shape how broadly to interpret the findings. FLORES-200 contains formal, Wikipedia-style text, so results for conversation and other writing styles remain unknown. Training samples were not equal across languages, and the multi-word stage used only part of the available sample because of memory limits. Code comparisons covered relatively small samples. The authors also note that some public competitor tokenizer files could not be verified against official releases.

Most importantly, the study measures compression—not language-model quality, actual API bills or serving speed. Fewer tokens could allow more text in a context window and fewer decoding steps for a fixed output, but the authors have not evaluated a model trained with v4 or v5. Tokenization changes how text is represented; it does not guarantee that a model learns from that representation equally well. The work is an arXiv preprint, and the evidence provided here does not establish its peer-review status. The paper and accompanying materials provide further methods and results.