VIGOR: Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward
Researchers have introduced VIGOR (Verifier-free Intrinsic Gradient-Norm Reward), a novel post-training paradigm for Large Language Models (LLMs) that eliminates the need for external verifiers or gold labels. Published on arXiv, this method leverages the policy model itself to generate intrinsic reward signals. VIGOR assigns higher rewards to completions with smaller l2 norms of teacher-forced negative log-likelihood gradients, indicating better alignment with the current policy. To ensure practical application, the approach corrects length bias using square root scaling and stabilizes reward scales through group-wise rank shaping. Experimental results demonstrate that VIGOR outperforms the state-of-the-art Reinforcement Learning from Internal Feedback (RLIF) baseline in mathematical reasoning tasks. Notably, it exhibits strong cross-domain transfer capabilities, improving code accuracy when trained solely on math data. For the Qwen2.5-7B-Base model, VIGOR achieved a 3.31% increase in math accuracy and a 1.91% increase in code accuracy compared to the baseline, while maintaining more stable training dynamics. The source code has been made publicly available to facilitate further research and adoption in scalable LLM optimization.
Wire timeline
VIGOR: Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward
Researchers have introduced VIGOR (Verifier-free Intrinsic Gradient-Norm Reward), a novel post-training paradigm for Large Language Models (LLMs) that eliminates the need for external verifiers or gold labels. Published on arXiv, this method leverages the policy model itself to generate intrinsic reward signals. VIGOR assigns higher rewards to completions with smaller l2 norms of teacher-forced negative log-likelihood gradients, indicating better alignment with the current policy. To ensure practical application, the approach corrects length bias using square root scaling and stabilizes reward scales through group-wise rank shaping. Experimental results demonstrate that VIGOR outperforms the state-of-the-art Reinforcement Learning from Internal Feedback (RLIF) baseline in mathematical reasoning tasks. Notably, it exhibits strong cross-domain transfer capabilities, improving code accuracy when trained solely on math data. For the Qwen2.5-7B-Base model, VIGOR achieved a 3.31% increase in math accuracy and a 1.91% increase in code accuracy compared to the baseline, while maintaining more stable training dynamics. The source code has been made publicly available to facilitate further research and adoption in scalable LLM optimization.
cs.AI updates on arXiv.org