Ai2 at COLM 2026: Olmo Hybrid Cuts Training Tokens by 49%, Bolmo Lands in Nature

Olmo Hybrid and the next Olmo

Ai2 presented the paper "Olmo Hybrid: From Theory to Practice and Back" at the conference. It looks at what happens when a language model combines transformer attention with linear recurrent layers. According to Ai2, attention can retrieve specific details from earlier text, while recurrent layers keep a compact state that updates as the model processes a sequence of text tokens.

The research links theory about what these architectures can represent with controlled training experiments. Ai2 says the study provides evidence that mixing attention and recurrence can improve training efficiency: Olmo Hybrid reached the same accuracy as Olmo 3 7B on MMLU, a benchmark covering knowledge across many subjects, using 49% fewer training tokens.

Ai2 says this work is informing the next Olmo model, which is now in pre-training with a hybrid mixture-of-experts architecture. Mixture-of-experts is another route to efficiency, routing each token through a subset of the model's components.

Open infrastructure and reproducibility

Ahead of COLM, Ai2 released Olmo-core 3, which adds a redesigned system for training large mixture-of-experts models. Researchers can use the code to develop their own models and investigate training decisions. Checkpoints and technical reports are available through the broader Olmo ecosystem.

Ai2 also pointed to a reproduction by Google. According to Ai2, Google recently announced that it recreated the Olmo 3 7B training run in MaxText on Google Cloud TPUs. Ai2 presents this as an example of the reproducibility its fully open approach enables.

At the conference, Noah A. Smith, Ai2's senior director of NLP research, spoke with communications lead Kyle Wiggers about the next iteration of Olmo and the lab's work on agentic models.

Asta and AI for science

Ai2 said scientific tools need to fit the tasks scientists actually do, which means building them with researchers and studying both their capabilities and limits. That is the approach it describes for Asta, its agentic platform for scientific work.

The recently released AstaBrief is an open-weights model that generates cited reports from research questions and retrieved literature. Scientists can use it in Asta's Fast mode or download it to run on their own infrastructure. Ai2 says it is also working directly with scientific communities to learn where general-purpose models fall short and what future open models should support. Senior research scientist Bodhisattwa Prasad Majumder attended COLM to continue those conversations and presented a number of related papers.

Bolmo published in Nature

During the conference, the paper "Retrofitting language models to operate over bytes", the research behind Bolmo, was published in Nature. Most language models first split text into chunks from a fixed vocabulary. Bolmo instead works with the underlying bytes that represent characters, which Ai2 says lets it handle details that subword tokenization can obscure.

The paper describes how to convert an existing language model to this approach with a relatively short additional training run. Alongside the publication, Ai2 released additional checkpoints so researchers can experiment with the architecture themselves.

What it means for researchers

The announcements point to Ai2's stated aim of releasing models, data, code and methods so others can test and extend its work. For researchers, the concrete pieces are the Olmo-core 3 training code, the Bolmo checkpoints and the open-weights AstaBrief model. Ai2 describes itself as a fully nonprofit lab and invites the community to read the research, try the models and share what they find.