Theoretical Analysis of Self-Training Algorithms in Deep Learning

This article from the Stanford AI Lab Blog presents a theoretical analysis of self-training, a key algorithmic paradigm in semi-supervised learning and domain adaptation. Addressing the challenge of scarce labeled data, self-training leverages abundant unlabeled data by using a pre-existing classifier to generate pseudo-labels, which are then used to retrain a new model. The authors explain why this counterintuitive process—retraining on predictions rather than true labels—provably improves accuracy over the original pseudo-labeler. A central finding is the critical role of regularization during the retraining phase; without it, minimizing cross-entropy loss fails to shift decision boundaries or enhance performance. The analysis highlights input consistency regularization, which encourages consistent predictions for neighboring examples, as an effective empirical technique. This post serves as the first part of a series examining methods that utilize unlabeled data, with subsequent discussions planned for self-supervised contrastive learning. The work aims to bridge the gap between recent empirical successes in unsupervised representation learning and rigorous theoretical understanding.