Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

Researchers have identified a new method to enhance the computational efficiency of Mixture of Experts (MoE) architectures in large language models by leveraging intra-expert activation sparsity. While MoE models typically rely on sparse expert activation, achieving finer granularity has been hindered by training challenges like expert collapse. This study reveals that existing pre-trained MoE models inherently possess substantial intra-expert sparsity, allowing up to 90% of neurons within each expert to be inactive without significant accuracy loss. The team analyzed eight off-the-shelf MoE models ranging from 1 billion to 400 billion parameters. By extending the vLLM execution pipeline to skip computations for these inactive neurons, they achieved a 2.5 times speedup in MoE layer execution and a 1.2 times end-to-end speedup compared to dense baselines. This approach requires no modifications to activation functions or model parameters, offering a practical solution for optimizing AI model performance.