Cartesia: Pretokenization Is a Hidden Prior That Shapes How Well Models Learn

The question: what do hand-built priors buy?

The post asks what the tradeoffs are between different approaches if the goal is to train the best language model possible with no priors, and how much those priors currently contribute. To answer it, Cartesia studies two kinds of scaling law. The first is compute-to-loss, the standard measure of how efficiently a model pretrains. The second is loss-to-loss, which relates pretraining strength (measured in loss or bits-per-byte) to downstream performance. Cartesia uses the loss-to-loss relationship as a proxy for how well a pretrained model actually understands its data.

The company compares a sequence of controlled models: a Byte Transformer, BPE with no pretokenization, BPE with pretokenization, H-Net, and BLT (Byte Latent Transformer). The aim is to see how decisions about processing input data affect performance.

A tension between efficiency and generalization

According to Cartesia, a central tension emerges. Standard language models working on compressed inputs have stronger compute efficiency, while models with access to finer-grained data, such as bytes, generalize better. The company says H-Nets get the best of both, and that it provides evidence their end-to-end nature is essential. It also reports a pattern in which the quality of a model's input processing seems strongly correlated with its ability to understand the data.

The post recalls results from the first part of the series: an H-Net's relative compute advantage grows from in-distribution pretraining loss to downstream distributions. Cartesia cites a 1.3x compute multiplier on a diverse data mixture, rising to 2x on downstream evaluations, and a 2x multiplier on the code subsplit, rising to 5x on out-of-domain languages.

Cartesia also challenges prior work, which stated that the tokenizer does not have a major impact on loss-to-loss laws (it cites LLMs on the Line, Figure 5). The company says it will provide empirical evidence that this belief is a myth.

Pretokenization as the hidden prior

The post spends time on pretokenization, a step that pre-segments the corpus with a hand-crafted regex before tokenization. Cartesia notes these regexes have grown steadily more complex and lists four examples: the GPT-2 pattern (also used in GPT-3, Falcon, StarCoder2 and EXAONE), cl100k (GPT-3.5 and GPT-4), o200k (GPT-4o, also used in gpt-oss and MiniMax-M1), and the DeepSeek-V3 pattern.

The changes between them include case-insensitive contractions, numbers split every three digits, words split on case so that CamelCase becomes Camel and Case, and, in DeepSeek-V3, a hardcoded CJK range covering ideographs and kana but not Korean.

The most obvious effect, Cartesia says, is breaking English text into whitespace-separated words. Without it, the vocabulary fills with tokens that carry little meaning, such as fragments of proper nouns like "North Kore" and "New Yor", or tokens that cross sentence punctuation, newlines and indentation. The company argues that pretokenizers are not learned from data, which runs counter to the bitter lesson, and that learning basic concepts such as words from bytes is a nontrivial problem.

Testing BPE without pretokenization

To isolate the effect, Cartesia trained a BPE tokenizer with no pretokenization, called BPE-NP, in-corpus using its usual methodology, and re-trained a scaling ladder. For the Byte Transformer, it trained new ladders from scratch with their own hyperparameter tuning. For every run, including those off the compute-optimal frontier, it plotted validation bits-per-byte against bits-per-byte on the OLMES evaluation suite. The runs fall on one smooth curve, which forms the ladder's loss-to-loss law.

Cartesia says BPE-NP shows an interesting tradeoff between compute-to-loss and loss-to-loss behavior, and that BPE tokenization noticeably improves on byte-level models in compute-to-loss terms. The provided text ends there, so the full results are not covered here.

What it means

The post suggests that decisions usually treated as preprocessing, such as tokenizer design and pretokenization regexes, may influence how well a model generalizes, not just how cheaply it trains. For anyone comparing architectures with scaling laws, Cartesia's argument is that loss-to-loss relationships may differ across input-processing methods and should not be assumed to transfer freely between model families.