PowerStep: Memory-Efficient Adaptive Optimization via l_p-Norm Steepest Descent
Researchers have introduced PowerStep, a novel memory-efficient optimizer designed for training large-scale neural networks like Transformers. While adaptive optimizers such as Adam are standard, they incur significant memory overhead by maintaining running estimates of gradient first and second moments. PowerStep addresses this by achieving coordinate-wise adaptivity without storing second-moment statistics, utilizing a nonlinear transform on a momentum buffer motivated by steepest descent under l_p-norm geometry. The authors prove that PowerStep converges at the optimal O(1/sqrt(T)) rate for non-convex stochastic optimization. Extensive experiments on Transformer models ranging from 124 million to 235 billion parameters demonstrate that PowerStep matches Adam's convergence speed while halving optimizer memory usage. Furthermore, when combined with aggressive int8 quantization, PowerStep remains numerically stable and reduces optimizer memory by approximately eight times compared to full-precision Adam. This development offers a principled, scalable, and resource-efficient alternative for large-scale AI training, with code made publicly available to support further research and adoption in the machine learning community.
Wire timeline
PowerStep: Memory-Efficient Adaptive Optimization via l_p-Norm Steepest Descent
Researchers have introduced PowerStep, a novel memory-efficient optimizer designed for training large-scale neural networks like Transformers. While adaptive optimizers such as Adam are standard, they incur significant memory overhead by maintaining running estimates of gradient first and second moments. PowerStep addresses this by achieving coordinate-wise adaptivity without storing second-moment statistics, utilizing a nonlinear transform on a momentum buffer motivated by steepest descent under l_p-norm geometry. The authors prove that PowerStep converges at the optimal O(1/sqrt(T)) rate for non-convex stochastic optimization. Extensive experiments on Transformer models ranging from 124 million to 235 billion parameters demonstrate that PowerStep matches Adam's convergence speed while halving optimizer memory usage. Furthermore, when combined with aggressive int8 quantization, PowerStep remains numerically stable and reduces optimizer memory by approximately eight times compared to full-precision Adam. This development offers a principled, scalable, and resource-efficient alternative for large-scale AI training, with code made publicly available to support further research and adoption in the machine learning community.
cs.AI updates on arXiv.org