A Hugging Face blog post from the Dharma-AI team describes VocabX, a method for specializing the tokenizer of small language models (SLMs) to a single domain or task. The authors argue that fine-tuned models inherit vocabularies built for many languages and text types, which creates two kinds of waste: domain-relevant sequences are fragmented into many tokens, and the model still scores thousands of vocabulary entries it almost never uses.
The team applied the approach to an automated Portuguese essay grader built on Qwen3-4B. The original tokenizer had 151,669 entries, and the model's generated text included repeated JSON envelopes, competency names, and evaluation phrases that could be represented more efficiently. VocabX kept the same base model, data, and training recipe while rebuilding the vocabulary around the cost of each token. The authors measured a 77% reduction in inference cost.
They also explain why reducing raw token count is not enough: in decode, each generated token needs a full pass through the network and is limited by memory bandwidth, making it roughly 7.4 times more expensive than a read token in their measured setup. The language-model head also reads a parameter row for every vocabulary entry at each generation step, so removing unused entries can cut cost even when token counts stay similar.
The numbers come from a paper that is still under review, so they may change before publication. The article does not compare VocabX with other vocabulary-specialization methods, and no independent verification is provided.