Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
Researchers have proposed a novel method called 'expert upcycling' to enhance the efficiency of training Mixture-of-Experts (MoE) large language models. While MoE architectures allow scaling model quality by increasing total parameters without raising per-token computation costs, training them remains expensive due to memory and communication demands. The new technique expands an existing trained model by duplicating experts and extending the router during continued pre-training. This approach provides a warm initialization, allowing the expanded model to start with lower loss compared to random initialization. Subsequent training breaks symmetry among duplicated experts, driving specialization. The study introduces utility-based expert selection using gradient-based importance scores, which significantly improves performance when training budgets are limited. Experiments scaling from 7B to 13B parameters demonstrated that upcycled models match the validation loss of fixed-size baselines while reducing GPU usage by 32%. This method offers a principled, compute-efficient alternative to training large MoE models from scratch, providing a practical recipe for deploying scalable AI systems with reduced resource consumption.
Wire timeline
Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
Researchers have proposed a novel method called 'expert upcycling' to enhance the efficiency of training Mixture-of-Experts (MoE) large language models. While MoE architectures allow scaling model quality by increasing total parameters without raising per-token computation costs, training them remains expensive due to memory and communication demands. The new technique expands an existing trained model by duplicating experts and extending the router during continued pre-training. This approach provides a warm initialization, allowing the expanded model to start with lower loss compared to random initialization. Subsequent training breaks symmetry among duplicated experts, driving specialization. The study introduces utility-based expert selection using gradient-based importance scores, which significantly improves performance when training budgets are limited. Experiments scaling from 7B to 13B parameters demonstrated that upcycled models match the validation loss of fixed-size baselines while reducing GPU usage by 32%. This method offers a principled, compute-efficient alternative to training large MoE models from scratch, providing a practical recipe for deploying scalable AI systems with reduced resource consumption.
cs.AI updates on arXiv.org