Q-learning with Adjoint Matching: A Novel RL Algorithm for Continuous Actions

Researchers Qiyang Li and Sergey Levine have proposed Q-learning with Adjoint Matching (QAM), a new temporal-difference-based reinforcement learning algorithm designed to address optimization challenges in continuous-action environments. The primary innovation of QAM lies in its ability to efficiently optimize expressive diffusion or flow-matching policies against a parameterized Q-function. Traditional methods struggle with numerical instability when backpropagating through multi-step denoising processes, often forcing compromises such as discarding gradient information or using biased approximations. QAM overcomes these limitations by employing adjoint matching, a technique from generative modeling that transforms the critic's action gradient into a stable, step-wise objective function. This approach ensures an unbiased and expressive policy at optimum without unstable backpropagation. When combined with temporal-difference backup for critic learning, QAM demonstrates superior performance compared to prior approaches on difficult, sparse reward tasks in both offline and offline-to-online reinforcement learning settings, marking a significant advancement in the field of machine learning and artificial intelligence.