Baseten Explains How to Pick Which Layers Run at 4 Bits in NVFP4

Why quantization helps

Quantization makes a model smaller and faster by storing its weights and activations in a lower-precision format. NVFP4 represents each value in 4 bits. Baseten says that with such coarse rounding, some layers lose information the model relies on, while others handle 4 bits with almost no decrease in output quality.

The post describes two kinds of workload. In memory-bound workloads, weights must be moved from VRAM into the GPU's compute units, so smaller weights mean fewer bytes to fetch. FP16 uses 2 bytes per weight, FP8 uses 1 byte and FP4 uses 0.5 bytes, which cuts fetched bytes by roughly 2x or 4x. In compute-bound workloads, the gain is arithmetic throughput: on hardware such as NVIDIA Blackwell GPUs with native FP4 support, FP4 tensor core operations can deliver up to 4x the peak FLOPS of FP16, according to the post.

Scales, blocks and calibration

FP4 can represent only 16 values: 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4 and ±6. Because real weights and activations are often much smaller, a scale maps that grid onto the range the values fall in. Weights and activations use separate scales.

With NVFP4, a tensor is split into blocks of 16 consecutive values, and each block sets its own scale. Baseten gives an example of why this matters: if one scale covered an entire array whose largest value is 84 while the rest sit around 1 or 2, the scale would become 84 ÷ 6 = 14. Every value below 3.5 would then be closer to 0 than to 7.

Weight scales can be computed directly because weights are fixed. Activations only exist once an input runs through the model, so their ranges are measured through calibration. This involves choosing a calibration dataset that reflects expected workloads such as coding or agentic tasks, recording activation distributions, choosing a clipping range, and computing a scale for each block of 16 values.

Three ways to choose a format per layer

Heuristic rules. These decide precision by architecture rather than by measuring sensitivity. A common rule quantizes MoE expert matrices to lower precision while keeping attention layers at higher precision, since attention is often more sensitive and expert matrices account for most of an MoE model's parameters. NVIDIA ModelOpt offers predefined configurations such as NVFP4_EXPERTS_ONLY_CFG, which quantizes only MoE expert layers, and NVFP4_MLP_ONLY_CFG, which quantizes MLP and MoE layers while leaving attention at higher precision.

Isolated-layer sensitivity scoring. ModelOpt AutoQuantize picks a format for each layer, such as NVFP4, FP8 or no quantization, by choosing the combination that minimizes estimated loss without exceeding a target memory size. It supports gradient-based, KL divergence and Aumann, Shapley scoring. The gradient method estimates how much loss would rise if one layer were quantized while every other layer stayed at baseline.

SaturationQuant. Baseten points to a limitation of isolated scoring: to estimate the loss from quantizing several layers at once, AutoQuantize cannot simply add up individual scores. SaturationQuant builds saturation into each score so the scores can be added across a mixed-precision recipe. Saturation means each layer adds less loss when others are quantized.

What it means for practitioners

The post frames NVFP4 quality as a per-layer allocation problem rather than a single switch. Teams relying on architecture-based presets skip sensitivity measurement entirely, while sensitivity-based tools need a calibration dataset and comparison against the full-precision model. Baseten argues that scores which can be added across layers give a more reliable way to compare mixed-precision recipes. The source text provided is cut off before the full SaturationQuant details, so results for the method are not covered here.