SimReg: Enhancing LLM Pretraining via Embedding Similarity Regularization

Researchers have introduced SimReg, a novel embedding similarity regularization loss designed to improve the pretraining efficiency of large language models (LLMs). While next-token prediction has driven significant advances in LLMs, it often results in high intra-class variance and inter-class similarity, hindering representation learning. SimReg addresses this by encouraging token representations with identical ground-truth labels to be more similar while enforcing separation from different-label tokens through contrastive loss. This mechanism enlarges multi-classification margins, facilitating more efficient classification. Extensive experiments conducted on both dense and Mixture-of-Experts (MoE) architectures demonstrate that SimReg accelerates training convergence by over 30% and boosts average zero-shot downstream performance by more than 1% across standard benchmarks. The study also provides practical insights into hyperparameter tuning and loss effectiveness through ablation studies. This work highlights the underexplored potential of similarity-based regularization in large-scale LLM pretraining, offering a promising direction for optimizing model performance and training speed in artificial intelligence development.