DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation

Researchers have introduced DARE, a novel framework designed to enhance the efficiency and effectiveness of reinforcement learning (RL) for large language models. While RL improves reasoning capabilities, it often suffers from high costs and sample inefficiency due to weak learning signals in many rollouts. Existing difficulty-aware data selection methods face limitations, including inaccurate estimates under policy drift and minimal gains in final performance or inference efficiency. DARE addresses these issues by co-evolving difficulty estimation with the policy using self-normalized importance sampling. It employs a symmetric Beta sampling distribution to ensure diverse difficulty coverage and applies adaptive compute allocation with tailored training strategies across different difficulty tiers. Extensive experiments demonstrate that DARE outperforms current methods in training efficiency, final effectiveness, and inference speed. The framework enables models to generate concise responses for easy tasks while significantly improving correctness on complex ones. This advancement suggests that efficient RL requires more than simple data filtering, necessitating a unified approach to difficulty estimation and policy learning. The associated code has been made publicly available to support further research and implementation in the field of artificial intelligence.