NVIDIA Cuts Cost of Bitwise-Deterministic Pretraining in Megatron Core to About 2%

What bitwise determinism means in Megatron Core

According to NVIDIA, bitwise determinism requires independent and checkpoint-resumed runs to follow the same numerical trajectory when data order, architecture, recipe, parallelism, software, runtime settings, and hardware are fixed. Megatron Core targets two guarantees.

• Independent reproducibility: two runs launched from the same initial state must remain bitwise identical at every training step.
• Bitwise checkpoint resume: a run that saves and restores one or more checkpoints must remain identical to an uninterrupted run.

NVIDIA argues the guarantee matters most for models with trillions of parameters trained across thousands of GPUs. There, multiple parallelism dimensions, low-precision computation, and distributed checkpointing make it harder to reproduce failures and validate fixes. Because production and hero training runs are expensive, the company says they need a strong guarantee of success. It also notes that at a trillion-parameter scale even a small slowdown can cost thousands of GPU-days, so deterministic training has to stay efficient.

Why teams want deterministic pretraining

NVIDIA lists four benefits. Teams can replay failures, for example by reproducing a loss spike at the same step and changing one component at a time to isolate the cause. They can preserve checkpoint continuity by resuming interrupted training without changing the numerical trajectory. They can detect unexpected differences, using mismatches between otherwise identical runs to investigate checkpoint corruption or software and hardware issues. And they can compare changes, separating the numerical effect of an intended change from run-to-run variation.

Verification needs more than watching the loss. NVIDIA points out that printed loss values are not sensitive enough, since two runs can print the same rounded loss while differing in the lower-order bits of their gradients or parameters. The recommended approach is to fingerprint selected metrics at every step, such as training loss, language-modeling loss, load-balancing loss, multi-token prediction (MTP) loss, and gradient norm, then compare two runs of the same configuration step by step. For checkpoint resume, the parameters, optimizer state, RNG state, data position, and subsequent outputs should match an uninterrupted reference after every restore.

Finding and fixing nondeterminism

At scale, NVIDIA says, nondeterminism can be intermittent or topology-dependent, and visible loss divergence can appear long after the first differing bit. The workflow proposed in Megatron-LM PR #7262 records ordered, per-rank tensor fingerprints for offline comparison.

The method narrows the search progressively, from the end-to-end metric to the training phase, module, operation, and finally the kernel. If two runs enter a scope with bitwise-identical inputs but leave it with different outputs, the divergence originated within that scope. Each selected rank writes its trace to a file without adding collectives or cross-rank ordering, which NVIDIA says reduces the risk of masking the race being investigated.

Fingerprints are computed on the GPU with torch.hash_tensor. For MXFP8 or NVFP4 tensors, both the encoded values and their scale buffers are fingerprinted. Order-sensitive data such as routing maps and MoE dispatch outputs should be fingerprinted by row or chunk, because whole-tensor XOR fingerprints cannot detect permutations. A matching fingerprint does not guarantee bitwise equality, so byte-level comparisons are advised for confirmation. The post also warns about false alarms: custom kernels can escape PyTorch dispatcher tracing, and uninitialized tensors, incomplete asynchronous collectives, and nonblocking copies may be read before their contents are valid.

What it means for large-scale training

According to NVIDIA, kernel-level changes such as giving grouped-GEMM writers private output slots and applying a fixed reduction order preserve parallelism while restoring deterministic results. Megatron Core also guards against regressions with recipe validation, kernel testing, and module validation across parallelism configurations including FP8 and FP4.

For teams running very large pretraining jobs, the claim is that reproducibility no longer has to come with a large speed penalty. NVIDIA suggests reviewing the Megatron Core User Guide for a supported recipe and comparing independent and checkpoint-resumed runs to validate reproducibility. The roughly 2% figure is NVIDIA's own measurement on its Nemotron workloads.