Hugging Face’s HOPE Method Prunes MoE Experts With Their Partners in Mind

Why pruning experts matters

Mixture-of-experts, or MoE, models can scale to hundreds of billions of parameters, but their experts create a substantial memory cost during inference. Hugging Face gives Qwen3.5-122B as an example: its 48 MoE layers each contain 256 experts, and 95% of its approximately 250 GB parameter memory footprint comes from the expert MLPs. For any individual token, however, only 8 experts in each layer are used.

Expert pruning permanently removes the least useful experts. According to Hugging Face, this can avoid the latency problems associated with dynamically loading or off-loading experts, requires no training, and can be combined with quantization. The challenge is choosing which experts to remove without damaging the model.

HOPE looks beyond individual expert scores

Existing pruning methods generally assign each expert an independent importance score and remove the lowest-scoring experts within each layer. Hugging Face calls these first-order methods because they treat each expert's contribution as independent.

The company says experts instead form stable cooperative groups that are repeatedly activated together. Removing one expert can therefore damage another expert that frequently works with it, even when the first expert appears unimportant by itself. This problem becomes more serious when pruning reaches high rates, such as 50% of the experts.

HOPE, short for Higher-Order Pruning of Experts, is designed as a second-order method. It uses pairwise interactions between experts to select a prune-set. Its objective is based on a surrogate for pruning error, with cross-terms that represent the gate-weighted output contributions of pairs of co-activated experts.

How the method works in practice

HOPE collects an interaction matrix from a calibration dataset. The matrix's diagonal represents individual expert importance, while its off-diagonal entries represent pairwise cooperative contributions. The method then formulates the selection of experts to remove as a quadratic program, using a continuous relaxation before rounding the result to binary solutions.

Hugging Face says HOPE uses the same calibration pass as other expert-pruning approaches, consisting of one forward pass over calibration data. The added quadratic-program step is non-convex but can be solved with a standard QP solver in 1, 2 seconds per layer. The company says calibration remains the dominant cost, making the additional overhead negligible compared with first-order methods.

At 50% pruning, Hugging Face reports that HOPE achieves the best average rank of any method it evaluated, with gains of up to +6.1% on agentic coding tasks over the next-best pruning method. The post also says HOPE's diagonal closely reproduces the expert rankings of REAP, a first-order pruning method, with a Spearman ρ of 0.988. That comparison is intended to show that HOPE's reported improvement comes from accounting for interactions, rather than from replacing individual expert importance scores.