Lila Sciences and Anyscale Train GLM-5.3-Flash on 1M-Token Sequences
Lila Sciences and Anyscale have trained the open model GLM-5.3-Flash on sequences of 1 million tokens using 64 B200 GPUs. The teams say the work extends their training stack to sequences 64 times longer than it could handle at the start of the project.
- GLM-5.3-Flash was trained on 1M-token sequences.
- The training used 64 B200 GPUs and the SkyRL open-source stack.
- The workload used 144.5 GiB per GPU at 1M tokens.
- Human-genome fine-tuning reduced held-out loss by 4.8% in 80 steps.
- Several fixes are being contributed upstream to Megatron-LM.
An open stack for long scientific data
The project targets a challenge in scientific AI: important structure can span a genome, related protein or RNA sequences, or an experiment conducted over several days. In an autonomous laboratory, context can also grow across hypotheses, protocols and instrument readouts to hundreds of thousands of tokens.
Many open models support 1M-token context, according to Anyscale, but may not deliver the required performance at that length without additional training. Lila Sciences and Anyscale trained GLM-5.3-Flash on task data at full length rather than cutting or heavily compacting the context. The work used SkyRL, an open-source training stack, with several fixes also headed upstream to Megatron-LM.
Memory and throughput were the main constraints
At 1M tokens per sequence, the workload used 144.5 GiB per GPU, below the B200's 178 GiB. The teams reported 13.7k tokens per second at 1M tokens and a peak of 24.9k tokens per second at 256k tokens on 64 B200 GPUs.
GLM-5.3-Flash combines 34 linear-attention layers using Kimi Delta Attention with 11 layers using DeepSeek Sparse Attention. It also has a residual stream four times wider than usual and mixture-of-experts layers. The researchers had to optimize each component because memory requirements increase with sequence length, while attention can grow quadratically.
Among the changes, they fixed sparse-attention layers that were silently falling back to dense attention, split scoring work across GPUs, and processed mixture-of-experts layers in sequence chunks. On real long-context data, the busiest GPU received about four times the average number of tokens, causing 512k-token runs to run out of memory when random-token tests did not.
Human-genome training showed long-context learning
The teams post-trained the model on the human genome using 512k-token sequences. Held-out loss fell 4.8% in 80 steps, with the largest improvement on tokens that had hundreds of thousands of tokens of preceding context.
Training used chromosomes 1 and 2 of the human reference genome, hg38, while evaluation used chromosome 22, which the model had not seen. Anyscale says this position-based result indicates the model learned to use long context rather than only local patterns.
All experiments ran on Ray through the Anyscale platform, on a 64-GPU B200 cluster in Lila's cloud. The teams say the open recipe is available in SkyRL, giving researchers a path to reproduce and extend the long-context training work.
