AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning

Researchers have introduced AdaGamma, a novel deep actor-critic method designed to enhance reinforcement learning through state-dependent discounting. While traditional deep RL methods typically employ a fixed discount factor across all states, this approach often limits adaptability. Although state-dependent discounting is theoretically attractive, naive implementations frequently suffer from instability and TD-error collapse. AdaGamma addresses these challenges by learning a state-dependent discount function alongside a return-consistency objective, which regularizes the induced backup structure. Theoretical analysis confirms the well-posedness of the Bellman operator under suitable conditions. Empirically, the method integrates seamlessly with popular algorithms like SAC and PPO, demonstrating consistent performance improvements on continuous-control benchmarks. Furthermore, AdaGamma achieved statistically significant gains in an online A/B test conducted on the JD Logistics platform. These findings suggest that coupling state-dependent discounting with return-consistency objectives effectively prevents degenerate target manipulation, making it a viable strategy for improving temporal adaptation in complex deep reinforcement learning environments.