Apple Researchers Train First Tri-Modal Masked Diffusion Model From Scratch

A different route to multimodal diffusion

According to the paper, discrete diffusion models have emerged as strong alternatives to autoregressive language models. Recent work has typically started from a base unimodal model and finetuned it for bi-modal generation. The Apple team says it diverges from that approach: instead of adapting an existing model, it pretrains the tri-modal MDMs from scratch on text, image-text and audio-text data.

The authors, a large group of Apple researchers with contributors affiliated with the University of Cambridge, MIT and Google DeepMind, note that some of them completed their work while at Apple. The study is listed under the research area Methods and Algorithms.

What the study examines

The team says it systematically analyzed several design choices for these models:

• multimodal scaling laws
• modality mixing ratios
• noise schedules
• batch-size effects

The paper also provides optimized inference sampling defaults. The authors describe their work as the largest scale systematic open study of multimodal discrete diffusion models conducted to date, and say it offers insights into scaling behaviors across multiple modalities.

A reparameterization for batch size

One technical result concerns batch size. According to the authors, their batch-size analysis yields a novel stochastic differential equation (SDE) based reparameterization. They say it removes the need to tune the optimal batch size, as reported in recent work.

The reparameterization decouples the physical batch size from the logical batch size. The physical batch size is often chosen based on compute considerations such as GPU saturation, FLOP-efficiency and wall-clock time. The logical batch size is chosen to balance the variance of gradients during stochastic optimization. Separating the two means each can be set for its own purpose.

A preliminary 3B model

To show what a unified design can do, the researchers pretrained a preliminary model. They report strong results at 3B model scale, trained on 6.4T tokens, across text generation, T2I tasks and T2S tasks. The source text does not provide benchmark figures or comparisons, so the strength of those results cannot be judged from the abstract alone.

What it means

For researchers working on multimodal generation, the paper offers a documented look at how scaling, data mixing, noise schedules and batch size interact when text, image and audio are trained together in a diffusion framework. The authors position the work as evidence that discrete diffusion can be built as a unified multimodal design from the start, rather than only through finetuning of unimodal models. The paper does not state any product plans or availability.