Cartesia Shows Hierarchical H-Nets Can Cut Compute Needs for Byte-Level Language Models
Cartesia says its hierarchical network architecture, H-Net, can replace tokenization in language models and deliver up to 3-5x compute efficiency gains in controlled tests. In the first of two posts, the company also reports that a 2-stage H-Net outperforms a matched open-source model with 10x less compute and data.
- Cartesia says 1-stage H-Nets deliver up to 3-5x compute efficiency gains by learning tokenization.
- A 2-stage H-Net reportedly outperforms a matched open-source model with 10x less compute and data.
- H-Nets chunk bytes dynamically inside the model instead of using offline tokenizers such as BPE.
- Cartesia plans to release the improved H-Net version and is exploring biology and vision uses.
- A second post will examine why H-Nets are so efficient for language modeling.
What H-Nets do
Hierarchical networks, or H-Nets, were introduced by Cartesia in 2025 as an end-to-end architecture that models multiple levels of abstraction in data. According to the company, the idea is to let a model recursively chunk low-level data into higher-level meaning: characters into words, words into phrases, phrases into concepts.
For language modeling, the practical effect is that H-Nets replace tokenization. Standard models rely on a pre-processing step that turns bytes into tokens, which are fixed chunks of bytes. H-Nets instead perform the chunking dynamically inside the model. Cartesia says the defining feature is that this happens hierarchically and end-to-end, whereas tokenized models and other alternatives use multiple training stages and do the chunking entirely offline.
The architecture has five modules: an encoder reads the inputs, a learned chunking step compresses them, a main network operates over the coarse chunks, the chunks are decompressed back to input resolution and summed with a residual, and a decoder finishes the processing. A 1-stage H-Net does one level of chunking, while the main network can itself be an H-Net, which allows nesting.
Why tokenization is the target
Cartesia frames the work around the bitter lesson, the idea from Richard Sutton's 2019 essay that general methods which scale with computation beat human-engineered priors. The company argues that language models largely follow this principle but still depend on hand-crafted steps such as tokenizers, which it says hinder end-to-end learning from the underlying data.
Models cannot read text directly and take tokens from a fixed vocabulary. Using the 256 byte values gives one token per UTF-8 byte, but those sequences are long and expensive to process. Byte Pair Encoding (BPE) addresses this by repeatedly merging the most frequent pair of adjacent tokens on a fixed corpus until the vocabulary reaches a set size, such as 128k for Llama-3.2. On Cartesia's data mix, each BPE token covers about 4.57 bytes. The company notes that this compression is fixed and that tokenization remains one of the last hand-designed parts of the pipeline.
Scaling ladder and reported results
The original H-Net work focused on language models of particular fixed sizes and data horizons, for example roughly 1.3B-parameter models trained on 100B tokens from FineWeb-Edu. Cartesia says a missing piece was compute-optimal scaling laws, now a standard way to measure pretraining innovations. As a reference point, it cites Kimi's work on Kimi Delta Attention (KDA) and attention residuals, which measured compute multipliers of 1.16x and 1.25x.
To test its improved architecture, Cartesia built a scaling ladder for a baseline BPE Transformer, following standard practices and reproducing results from the literature. It matched the training mixture and re-trained a BPE tokenizer for the baseline. It then converted each rung into a byte-level H-Net as a drop-in replacement, reusing the BPE backbone architecture and hyperparameters. According to the company, this recipe showed stronger validation loss (1.3x).
Cartesia reports that carefully controlled 1-stage H-Nets give compute efficiency gains of up to 3-5x purely through learned tokenization. It also says a more flexible 2-stage H-Net outperforms a matched open-source model with 10x less compute and data.
Beyond text and what comes next
The company says it analyzed the patterns H-Nets learn and found behaviors not previously seen in language models, such as in-context tokenization. It also points to ongoing research in biology and vision, and argues that H-Nets can operate on any type of data, including continuous data, where tokenization is a serious challenge.
Cartesia says the H-Net version used in these results is much improved over the earlier one and that it plans to release it. A second post will examine a larger range of models to explain why H-Nets can be so much more efficient for language modeling. All figures here are Cartesia's own claims from its blog post, and the source text provided does not include full benchmark details.
Cartesia: From Tokens to Bytes I: Scaling Hierarchical Networks | Cartesia
