VocabX: Specializing an SLM's Vocabulary Cuts Inference Cost by 77%
Dharma-AI says it cut measured inference cost by 77% in a production essay grader without changing the base model, by rebuilding the tokenizer vocabulary around the cost of each token rather than its frequency. The method, called VocabX, is described in a Hugging Face blog post that previews a paper the team says it will publish soon.
- Dharma-AI says VocabX cut measured inference cost by 77% on a Qwen3-4B Portuguese essay grader, with the same base model, data and training recipe.
- The method selects tokens by cost, not frequency: in the grader, a generated token costs 7.4 times a token read.
- VocabX works on sub-word and multi-word (macro token) layers under one cost criterion.
- The grader's output layer shrank from about 152,000 to about 65,000 rows.
- The figures come from a paper still under review that may change before publication.
A tokenizer built for everyone, used for one task
The case study is an automated grader for Portuguese essays, built on Qwen3-4B. According to the team, it receives the essay, the prompt and the supporting texts, and returns a JSON assessment with a grade and a comment for each competency. The model was already small, at 4 billion parameters, and already trained for the task.
Its tokenizer, however, was still the original Qwen3-4B one, built for many languages and many kinds of text. The authors say this created two kinds of waste. The first is fragmentation: the tokenizer splits into many pieces exactly what the domain repeats most, such as the JSON envelope, the competency names and the evaluation phrases that start every response. The second is excess: the vocabulary of 151,669 entries includes many that the task almost never uses, yet the model scores all of them at every generated token.
The team kept the same base model, the same data and the same training recipe, and changed only the vocabulary. They note that the numbers come from the current version of the paper, which is still under review and may change before publication.
Why generated tokens cost more than read ones
The core argument is that counting tokens is not the same as measuring the bill. Inference has two phases. In prefill, the model processes the whole input in one pass and the limit is compute. In decode, it generates the response one token at a time, and the limit becomes memory bandwidth, because the GPU spends most of its time waiting for data from memory. According to the authors, this effect is strongest for small models served with moderate batches.
In the grader they measured, a generated token costs 7.4 times a token read. The authors stress that this ratio varies with hardware, batch size, context length and other factors, and can be larger, but that generating generally costs more than reading.
The vocabulary adds a third cost. The LM head holds one row of parameters per vocabulary entry and, in a model the size of Qwen3-4B, accounts for almost 10% of the parameters. It reads and multiplies the entire matrix for every generated token, so an unused entry costs as much as a frequent one. In the grader, the output layer went from about 152,000 to about 65,000 rows.
How VocabX works
The authors say existing approaches such as fast vocabulary transfer (FVT), multi-word tokenization (MWT) and Liquid AI's in-place BPE already reduce fragmentation, but they choose tokens by frequency, not cost, and work on a single layer. VocabX changes this in three ways.
- Cost-based selection. Each eliminated token is multiplied by the price of the field it appears in: system prompt, user message or model response. In the grader, a token saved in the response is worth 7.4 times one saved in the input.
- Two layers under one criterion. At the sub-word layer, it rebuilds how domain words are cut. At the multi-word layer, it turns repeated spans, from short expressions to full lines and JSON envelopes, into single macro tokens. Each layer has its own budget.
- Estimating what the sample cannot see. Choosing by cost requires knowing how much each candidate still saves after earlier choices. The source text is cut off while describing this third step.
To find repeated spans, the team uses five miners, each specialized in one kind of repetition: long words, whole lines, expressions, complete sentences, and structures with punctuation such as JSON and tags. Averaged over eight corpora, the last one contributed the most, and on code it alone accounted for 17.5 points. Without all five, token reduction on the grader would drop by 55.6 points.
What it means for specialized models
The post argues that the tokenizer, usually treated as a fixed property of a base model, can be one of the largest cost levers for a model specialized in a single domain, and that it is set before any training step. The same logic, the authors say, applies to contracts with boilerplate clauses or code with repeated identifiers.
Teams that fine-tune open models such as Qwen and Gemma for narrow tasks may therefore find savings in the vocabulary rather than in the model size. The 77% figure comes from one production system and a paper still under review, so results elsewhere may differ.
