Vector Institute Research Reveals SGD Outperforms Adaptive Optimizers in Neural Network Robustness
Researchers from the Vector Institute for Artificial Intelligence have published findings indicating that neural networks trained using Stochastic Gradient Descent (SGD) exhibit significantly higher robustness to input perturbations compared to those trained with adaptive gradient optimizers like Adam and RMSProp. While standard test accuracy remains comparable across these optimization methods, the study highlights a critical vulnerability in adaptive methods when facing unexpected changes in input data. Through empirical and theoretical analyses, including frequency-domain investigations, the authors demonstrate that adaptive optimizers may allow models to rely on irrelevant data frequencies, making them more sensitive to disturbances. This research, featured in TMLR, suggests that choosing SGD over adaptive methods can enhance the reliability of machine learning systems. The findings are particularly relevant for developers aiming to build resilient AI models capable of maintaining performance under various attack vectors, such as Gaussian noise or bounded attacks. The team has released accompanying code to support further exploration of these optimization dynamics, emphasizing the trade-off between standard generalization and robustness in deep learning training strategies.
Wire timeline
Vector Institute Research Reveals SGD Outperforms Adaptive Optimizers in Neural Network Robustness
Researchers from the Vector Institute for Artificial Intelligence have published findings indicating that neural networks trained using Stochastic Gradient Descent (SGD) exhibit significantly higher robustness to input perturbations compared to those trained with adaptive gradient optimizers like Adam and RMSProp. While standard test accuracy remains comparable across these optimization methods, the study highlights a critical vulnerability in adaptive methods when facing unexpected changes in input data. Through empirical and theoretical analyses, including frequency-domain investigations, the authors demonstrate that adaptive optimizers may allow models to rely on irrelevant data frequencies, making them more sensitive to disturbances. This research, featured in TMLR, suggests that choosing SGD over adaptive methods can enhance the reliability of machine learning systems. The findings are particularly relevant for developers aiming to build resilient AI models capable of maintaining performance under various attack vectors, such as Gaussian noise or bounded attacks. The team has released accompanying code to support further exploration of these optimization dynamics, emphasizing the trade-off between standard generalization and robustness in deep learning training strategies.
Vector Institute for Artificial Intelligence