Quantile Geometry Regularization for Distributional Reinforcement Learning

Researchers have introduced Robust Quantile-based Implicit Quantile Networks (RQIQN), a novel enhancement for distributional reinforcement learning aimed at resolving issues with distorted or degenerate distribution estimates. Traditional quantile-based methods often suffer from bootstrapped target quantiles that lead to inaccurate return distributions. The proposed RQIQN approach reinterprets the Implicit Quantile Network (IQN) loss as local empirical quantile estimation problems. By applying a Wasserstein distributionally robust formulation, the method generates a closed-form, fraction-dependent correction to the Bellman target. This correction preserves the risk-neutral quantile average through median antisymmetry while enlarging upper-lower quantile gaps to prevent collapsed distributional spread. Crucially, RQIQN regularizes quantile geometry without altering the underlying value objective or requiring additional sample set reconstruction. Empirical evaluations demonstrate that RQIQN outperforms existing quantile-based algorithms in risk-sensitive navigation tasks and Atari games, marking a significant advancement in robust reinforcement learning techniques for complex environments.