Modal shows stable RL training with mixed-precision, mixed-accelerator samplers

Why mixed samplers

According to Modal, samplers in large-model RL usually run on the same accelerators, and often the same precision, as the trainer, which keeps the two numerically close and training stable. Serving systems outside RL face no such limit. They can use low-precision formats such as FP8, MXFP4 and NVFP4, co-designed kernels, and even accelerators built for long-context prefill.

Modal notes that sampling already takes 80 to 85% of the GPUs in large RL runs, so restricting samplers leaves savings unused. The company asks whether samplers with different precisions, accelerators and kernels can feed one trainer while keeping training stable and performant.

It names two obstacles. The first is learning: each optimization adds training-inference mismatch, making samples more off-policy, which often destabilizes training. The second is the system: RL infrastructure is usually built around homogeneous clusters, while mixed samplers need accelerators that scale up and down on demand.

The experimental setup

Modal trained Qwen3.6-35B-A3B on agentic coding with up to 256K tokens of context, using the 2.7K repository tasks Xiaomi released with MiMo-V2.6. The company removed repository history and other traces through which some tasks leak the reference fix. In each episode the model writes a patch over multiple turns and submits it through a bash tool in a Modal Sandbox. The reward is 1 if the patch passes the task's tests in a fresh sandbox, and 0 otherwise. Evaluation uses SWE-bench Pro V2, which has no overlap with the training set, and its hard subset HARD-51. Every checkpoint is evaluated in BF16.

The framework is Miles, with SGLang samplers and a Megatron-LM trainer that updates the policy in BF16 on B200 GPUs. Training is fully asynchronous with no limit on staleness, and each update uses 128 prompts with 8 trajectories each. The trainer does not replay the samplers' MoE routing decisions.

The mixed samplers span six accelerator types. B200 and B300 serve NVFP4 weights, H100 and H200 serve FP8 and BF16 weights, and A100 and RTX PRO 6000 serve BF16 weights. Each runs its own kernels and keeps its KV cache in FP8. A router assigns each agent session to one of eight groups in proportion to serving capacity. The homogeneous baseline runs only on B200 with BF16 weights and a BF16 KV cache.

What keeps training stable

Modal starts from GRPO without correction and adds three pieces:

• Masked importance sampling (MIS) multiplies each token's term by its importance ratio and masks tokens whose ratio falls outside a fixed range, keeping ratios bounded.
• Score centering (SC) removes drift by subtracting the expected gradient of the log-probability under the sampling policy.
• Top-p sampling with mask replay samples only from the top-p candidate set, and the trainer renormalizes its probabilities over the same set, so rare tokens with extreme ratios never enter the data.

Plain GRPO collapses on both setups. On the mixed samplers it collapses soon after training starts, while on the homogeneous samplers it lasts longer before collapsing as well. Each correction alone only delays the collapse, and every such run ends at or below the base model. Only SC + MIS, with or without top-p, trains stably to the end. Pass@4 rises over roughly the first 100 to 140 steps, then levels off at about 65 to 75% on HARD-51. On the full set it goes from 89% to about 94%.

Modal reports that every collapse shares three signs: training reward falls, sampler-trainer KL grows 3× or more, and most episodes end on repeated format errors. In collapsed episodes the model stops calling its tool partway through a task. At GRPO's step-40 checkpoint on the mixed samplers, 62% of evaluation episodes end this way, a median of 22 turns in, against 57 turns for passing episodes.

Parity with homogeneous samplers, at lower cost

With SC + MIS + top-p, the mixed samplers reach SWE-bench Pro V2 scores on par with the homogeneous samplers, and the two runs' pass@4 stays close throughout training. Training reward and gradient norm follow similar paths, despite a 4.7× larger sampler-trainer KL on average. Entropy matches until about step 160, then rises on the homogeneous samplers and stays flat on the mixed ones.

For most of training, the samplers' cost per update was about 15% lower on the mixed samplers. Modal says its setup was built to validate performance rather than minimize cost, so further savings are possible by tuning each sampler and shifting the mix toward the most cost-effective ones.

What it means

The results suggest RL teams need not tie sampling to the trainer's hardware and precision, provided they apply off-policy correction and can scale heterogeneous accelerators on demand. Modal says nothing in the correction is specific to this model or these accelerators, so it expects the pattern to hold more broadly. The evidence so far comes from a single model and task family.