Why Adam Works Better with β1 = β2: The Missing Gradient Scale Invariance Principle
Researchers from arXiv have published a study explaining why the Adam optimizer performs better when its momentum parameters are equal (β1 = β2). Despite Adam's decade-long dominance in large-scale machine learning training, this empirical observation lacked a theoretical basis. The authors introduce and formalize the concept of 'gradient scale invariance,' proving that Adam achieves first-order gradient scale invariance if and only if β1 equals β2. This structural property aligns Adam with recent optimizers designed for scale-robust updates. The theory is validated through extensive experiments across various computer vision and natural language processing tasks, demonstrating that gradient rescaling results in smoother update effects under this balanced regime. These findings provide a coherent explanation for an open question in optimization behavior and offer a guiding principle for designing future robust optimizers. The paper contributes significantly to the understanding of deep learning optimization dynamics, bridging the gap between empirical practice and theoretical justification in artificial intelligence research.
Wire timeline
Why Adam Works Better with β1 = β2: The Missing Gradient Scale Invariance Principle
Researchers from arXiv have published a study explaining why the Adam optimizer performs better when its momentum parameters are equal (β1 = β2). Despite Adam's decade-long dominance in large-scale machine learning training, this empirical observation lacked a theoretical basis. The authors introduce and formalize the concept of 'gradient scale invariance,' proving that Adam achieves first-order gradient scale invariance if and only if β1 equals β2. This structural property aligns Adam with recent optimizers designed for scale-robust updates. The theory is validated through extensive experiments across various computer vision and natural language processing tasks, demonstrating that gradient rescaling results in smoother update effects under this balanced regime. These findings provide a coherent explanation for an open question in optimization behavior and offer a guiding principle for designing future robust optimizers. The paper contributes significantly to the understanding of deep learning optimization dynamics, bridging the gap between empirical practice and theoretical justification in artificial intelligence research.
cs.AI updates on arXiv.org