Your Experts Have Partners. There's HOPE for Keeping Them Together

What happened

A group of researchers introduced HOPE, a new pruning technique for Mixture-of-Experts (MoE) language models. HOPE is designed to reduce the memory required by these models by removing certain components, called experts, while taking into account how these components interact with each other. The method was tested and found to maintain or improve performance, especially on coding tasks, even when half of the experts were removed. The results showed that HOPE performed better than previous pruning methods in these scenarios.

Key details

• HOPE stands for Higher-Order Pruning of Experts.
• The method considers pairwise interactions between experts when deciding which to prune.
• At 50% pruning, HOPE improved coding task accuracy by up to 6.1% compared to the next-best method.
• Mixture-of-Experts models like Qwen3.5-122B can have hundreds of experts per layer, with most inactive for any given input.
• HOPE does not require retraining the model after pruning.

Why it matters

Developers and organizations using large MoE language models can now reduce the memory needed to run these models without losing accuracy, especially for tasks like code generation. This can lower hardware costs and make it possible to deploy powerful models in more resource-constrained environments. Model maintainers can use HOPE to shrink models efficiently without the need for additional training or complex loading strategies.