Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
A new research paper published on arXiv investigates the impact of curriculum learning on the pretraining dynamics of Large Language Models (LLMs). The study, conducted by Mohamed Elgaar and Hadi Amiri, examines how ordering pretraining data affects model training across scales ranging from 14 million to 1 billion parameters. The researchers tested three linguistically motivated curricula—Age-of-Acquisition, word frequency, and Verb Variation—against random data ordering. Key findings indicate that while training follows a shared sequence of latent phases regardless of method, curricula primarily influence the duration spent in each phase. Random ordering resulted in higher gradient noise scale and output-head saturation in smaller models, whereas structured curricula promoted more stable within-phase training. Notably, the direction of the curriculum matters, as reverse-ordering reduced accuracy advantages. These stability differences diminish at larger model scales. The results suggest that curriculum learning enhances training stability in smaller models rather than creating new learning phases, offering valuable insights for optimizing LLM pretraining strategies and data organization.
Wire timeline
Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
A new research paper published on arXiv investigates the impact of curriculum learning on the pretraining dynamics of Large Language Models (LLMs). The study, conducted by Mohamed Elgaar and Hadi Amiri, examines how ordering pretraining data affects model training across scales ranging from 14 million to 1 billion parameters. The researchers tested three linguistically motivated curricula—Age-of-Acquisition, word frequency, and Verb Variation—against random data ordering. Key findings indicate that while training follows a shared sequence of latent phases regardless of method, curricula primarily influence the duration spent in each phase. Random ordering resulted in higher gradient noise scale and output-head saturation in smaller models, whereas structured curricula promoted more stable within-phase training. Notably, the direction of the curriculum matters, as reverse-ordering reduced accuracy advantages. These stability differences diminish at larger model scales. The results suggest that curriculum learning enhances training stability in smaller models rather than creating new learning phases, offering valuable insights for optimizing LLM pretraining strategies and data organization.
cs.AI updates on arXiv.org