Flow Matching Enhances TD Learning in Reinforcement Learning

A new research paper titled 'What Does Flow Matching Bring To TD Learning?' investigates the effectiveness of flow matching for scalar Q-value function estimation in reinforcement learning (RL). Contrary to conventional beliefs that attribute its success to distributional RL, the authors demonstrate that explicitly modeling return distributions can actually reduce performance. Instead, they argue that flow matching improves Temporal Difference (TD) learning through two key mechanisms: test-time recovery and plastic feature learning. Test-time recovery allows iterative computation via integration to dampen errors in early value estimates, a capability absent in monolithic critics. Additionally, supervising the velocity field at multiple interpolant values induces more plastic feature learning, enabling critics to handle non-stationary TD targets without overfitting or discarding previously learned features. Empirical validation shows that flow-matching critics outperform monolithic critics by twofold in final performance and fivefold in sample efficiency, particularly in high-UTD online RL problems where loss of plasticity is a challenge. This study provides a formal explanation for the advantages of flow matching in RL contexts.