TRACE: Token-Routed Self On-Policy Alignment for Critical Reasoning
Researchers have introduced TRACE (Token-Routed Alignment for Critical rEasoning), a novel method designed to enhance on-policy self-distillation in reinforcement learning with verifiable rewards (RLVR). The study identifies that traditional all-token KL divergence methods waste gradients on redundant positions and cause privileged-information leakage, leading to performance degradation in long-horizon math tasks. TRACE addresses this by distilling only on annotator-marked critical spans, utilizing forward KL on key spans of correct rollouts and optional reverse KL on error spans, while applying GRPO to remaining tokens. Experimental results on four math benchmarks and GPQA-Diamond show that TRACE improves performance by an average of 2.76 percentage points over GRPO. Notably, it preserves out-of-distribution scores where baseline methods fail. The approach remains effective even with online self-annotation, achieving approximately 69% of the gain provided by strong external APIs. The optimal routing strategy varies by model scale, with forward KL suiting larger models like Qwen3-8B and reverse KL benefiting smaller ones like Qwen3-1.7B.
Wire timeline
TRACE: Token-Routed Self On-Policy Alignment for Critical Reasoning
Researchers have introduced TRACE (Token-Routed Alignment for Critical rEasoning), a novel method designed to enhance on-policy self-distillation in reinforcement learning with verifiable rewards (RLVR). The study identifies that traditional all-token KL divergence methods waste gradients on redundant positions and cause privileged-information leakage, leading to performance degradation in long-horizon math tasks. TRACE addresses this by distilling only on annotator-marked critical spans, utilizing forward KL on key spans of correct rollouts and optional reverse KL on error spans, while applying GRPO to remaining tokens. Experimental results on four math benchmarks and GPQA-Diamond show that TRACE improves performance by an average of 2.76 percentage points over GRPO. Notably, it preserves out-of-distribution scores where baseline methods fail. The approach remains effective even with online self-annotation, achieving approximately 69% of the gain provided by strong external APIs. The optimal routing strategy varies by model scale, with forward KL suiting larger models like Qwen3-8B and reverse KL benefiting smaller ones like Qwen3-1.7B.
cs.AI updates on arXiv.org