Artificial intelligence news.
Straight from the original announcements.

PyTorch and TorchTitan Train a 1T-Parameter MoE on JUPITER

November 12, 2026, 13:00By the AI News Bites Editorial TeamSource: PyTorch
PyTorch and TorchTitan Train a 1T-Parameter MoE on JUPITER
Image: PyTorch

The EUROPA consortium says it has scaled pretraining for a roughly 1-trillion-parameter Mixture-of-Experts model on JUPITER, a publicly funded European supercomputer. Using PyTorch and TorchTitan, the team increased throughput from 90 to approximately 1,300 tokens per second per GPU across 384 NVIDIA GH200 GPUs.

Key points
  • EUROPA trained a 951.9B-parameter MoE on the public JUPITER supercomputer.
  • The run used 384 NVIDIA GH200 GPUs across 96 nodes.
  • Throughput increased from 90 to approximately 1,300 tokens per second per GPU.
  • FlexShard reduced communication for the Muon optimizer.
  • Activation offloading moved checkpoint data from HBM to Grace CPU memory.

Frontier-scale training on shared infrastructure

The work was carried out as part of the EuroHPC Frontier AI Grand Challenge, which was designed to test whether sovereign, open-source AI could reach frontier levels using shared public resources. The EUROPA consortium, formed by Domyn and Fraunhofer IAIS and IIS, was selected to develop an open European MoE model with at least 400B parameters, support for all 24 official EU languages and compliance with the EU AI Act.

The training ran on JUPITER, the EuroHPC exascale supercomputer hosted at Forschungszentrum Jülich. Unlike the private clusters usually used for models of this size, JUPITER is public research infrastructure shared by hundreds of projects. The consortium says this made the system's architectural constraints central to the engineering effort.

A 951.9B-parameter model and custom parallelism

The configuration described in the post contains 951.9B total parameters, with 32.8B active per token. It has 74 layers, a hidden size of 6,144, 384 routed experts with top-8 routing, an expert hidden size of 1,792 and a vocabulary of 256,000. The pretraining sequence length was 65,536 tokens, using Muon and AdamW as optimizers.

Each JUPITER Booster node contains four NVIDIA GH200 Grace Hopper Superchips. Every superchip combines a Hopper GPU with 96 GB of HBM3 and a 72-core Grace CPU with 120 GB of LPDDR5X, connected by a 900 GB/s NVLink-C2C link. The consortium used 384-way FSDP, four-way expert parallelism within each node and no pipeline parallelism.

The layout reflects the system's communications limits. Inter-node InfiniBand provides about 25 GB/s per GPU, while the GPU-to-CPU link inside a superchip reaches 900 GB/s. The team therefore kept the MoE token all-to-all on NVLink and worked to overlap communication with computation.

PyTorch and TorchTitan changes improved throughput

The consortium selected PyTorch and TorchTitan for the flexibility needed to adapt the pretraining system. One key change was FlexShard, which controls parameter ownership for the Muon optimizer. Sharding experts by whole expert allowed ranks to run Newton-Schulz iterations without communication for those matrices. Running the optimizer on the GPU while parameters remained offloaded to Grace reduced optimizer time from 41.7 seconds to 2.2 seconds, increasing throughput from 441 to 597 tokens per second per GPU.

The team also added activation-checkpoint offloading to Grace. At 65,536 tokens, boundary activations required approximately 55 GiB per GPU, so storing them in CPU memory freed HBM for other model state. Throughput increased from 370 to 508 tokens per second per GPU with offloading, while the sequence length grew first to 49,152 tokens and then to 65,536 tokens with Muon.

According to PyTorch, the combined work raised throughput from 90 to approximately 1,300 tokens per second per GPU on 384 GH200 GPUs across 96 nodes. The project also contributed FlexShard, activation-checkpoint offloading hooks and other changes back to TorchTitan and PyTorch, giving users of those projects access to techniques developed for public, large-scale infrastructure.

Related news

PyTorch team integrates SGLang with IBM Spyre in less than a week
For developersNovember 12, 2026, 15:45

PyTorch team integrates SGLang with IBM Spyre in less than a week

A two-person team integrated SGLang with IBM Spyre in less than a week, using AI coding agents to build an open source plugin of roughly 1,500 lines. PyTorch says SGLang…

PyTorch serves Meta’s 816B Muse Spark Open on one TPU v7 host
For developersNovember 10, 2026, 17:45

PyTorch serves Meta’s 816B Muse Spark Open on one TPU v7 host

PyTorch says it can serve Meta’s 816B-parameter Muse Spark Open mixture-of-experts model on a single TPU v7 host containing four chips. The system uses SGLang-TorchTPU…

Google brings long-context training to TPUs through MaxText
For developersNovember 9, 2026

Google brings long-context training to TPUs through MaxText

Google has added long-context training support to MaxText, its open-source JAX training framework for Cloud TPU. The release supports four context-parallel strategies…