Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

A new research paper published on arXiv investigates methods to enhance the effectiveness of Reinforcement Learning (RL) in Large Language Models (LLMs). The study addresses the limitation that RL performance often suffers when training data lacks diversity in reasoning approaches. To solve this, the authors propose a mid-training phase using diverse, self-generated data before the final RL stage. They employ a bootstrapped data-generation framework inspired by George Polya's problem-solving strategies to create multiple variants of correct answers for each training question. Theoretical analysis suggests that policy-gradient updates incentivize the combination of these varied approaches. Empirical results demonstrate that LLMs initialized with this mid-training data achieve consistent improvements across mathematical reasoning benchmarks and out-of-distribution tasks, such as code generation and narrative reasoning. The findings indicate that exposing language models to multiple problem-solving pathways through self-generated data significantly benefits subsequent reinforcement learning processes, offering a promising direction for improving AI reasoning capabilities.