Ai2's Bolmo byte-level language models get Nature paper and new checkpoints

From Olmo to other model families

Ai2 introduced Bolmo last December. According to the institute, the Nature paper shows that its conversion method, which it calls byteifying, is not limited to the Olmo models that served as the starting point for Bolmo.

Byteifying takes an already capable subword model and turns it into a byte-level one with a relatively short additional training run. Ai2 used it to build Bolmo 1B and Bolmo 7B from its open Olmo models. The institute says it believes these are the first fully open byte-level language models that are competitive with strong subword models across a broad range of tasks.

In the paper, Ai2 applied the same process to Qwen 3 8B and Llama 3 8B, creating Bwen 8B and Blama 8B. Ai2 says both come close to matching the models they were derived from. Bwen 8B is described as the institute's strongest byteified model so far, outperforming Bolmo 7B across its aggregate evaluation suite.

Ai2 is also releasing its Stage 1 checkpoints. These keep the original global model frozen while the new byte-level components of Bolmo are trained, which the institute says gives researchers a faster starting point for experimenting with and extending the architecture.

Why bytes instead of subwords

Most language models do not process text directly. They first split it into subwords, which are word chunks drawn from a fixed vocabulary. Ai2 says this works remarkably well, but it can make models less flexible with spelling, unusual strings, rare words, or text that does not map cleanly onto the predefined chunks.

Byte-level models instead work on the underlying bytes that computers use to represent characters. According to Ai2, this can improve a model's understanding of fine-grained text structure, from whitespace to different writing systems, without tying it to a fixed vocabulary. The institute also points out that bytes are a fundamental representation of digital data, so byte-level modeling could eventually offer a common foundation for other kinds of information, including images and audio.

The obstacle has been cost. Ai2 notes that matching the performance of subword models has historically required training byte-level models from scratch, which makes it hard for them to keep pace as subword models improve quickly. Byteifying is meant to avoid that expense.

Early use and what comes next

Ai2 says developers and researchers have already begun adapting Bolmo for specific applications. One example is a Bolmo 7B fine-tune for Russian-to-English poetry and lyric translation. Researchers are also using Bolmo as a base for experiments on transferring capabilities between byte-level models, and for studies of the tradeoff between the efficiency of hierarchical byte architectures like Bolmo and fine-grained character understanding.

The Bolmo publication follows other recent Ai2 work in Nature. In February, the institute published a paper titled "Synthesizing scientific literature with retrieval-augmented language models", which showed how specialized retrieval, ranking, and citation handling can help language models synthesize scientific literature with more reliable grounding.

Ai2 argues that fundamental advances in AI are more impactful when the research community can inspect, reproduce, and build on them. For Bolmo, it says this opens a path to models that are not locked into a single way of representing information. The institute suggests future systems could adapt their representations across languages, domains, and tasks, while researchers test new architectures without repeating the full cost of training. By making Bolmo and its training recipe fully open, Ai2 says it hopes to help researchers challenge assumptions built into today's language models.