Revisiting LDA on Frozen CNN Features for Supervised Dimensionality Reduction
A new study published on arXiv reevaluates the use of Linear Discriminant Analysis (LDA) for supervised dimensionality reduction in transfer learning pipelines utilizing frozen pretrained image representations. The research investigates whether applying LDA before linear probing offers advantages over using full feature vectors, particularly when target tasks have fewer classes than pretraining tasks. Evaluating ten dimensionality-reduction strategies across six backbone architectures—including ResNet, MobileNet, EfficientNet, and ViT variants—on datasets like CIFAR-100, Tiny ImageNet, and CUB-200-2011, the authors found significant benefits for coarse-grained classification. LDA improved accuracy by up to 4.5 percentage points while reducing feature dimensionality by 48-87% in most coarse-grained scenarios. However, the method consistently underperformed on fine-grained tasks such as CUB-200, where subtle cues are critical. The study concludes that plain LDA provides the optimal accuracy-cost tradeoff for coarse-grained settings, whereas complex supervised reduction methods rarely justify their additional computational costs. These findings offer practical guidance for integrating post-hoc supervised projections into frozen-feature image classification systems.
Wire timeline
Revisiting LDA on Frozen CNN Features for Supervised Dimensionality Reduction
A new study published on arXiv reevaluates the use of Linear Discriminant Analysis (LDA) for supervised dimensionality reduction in transfer learning pipelines utilizing frozen pretrained image representations. The research investigates whether applying LDA before linear probing offers advantages over using full feature vectors, particularly when target tasks have fewer classes than pretraining tasks. Evaluating ten dimensionality-reduction strategies across six backbone architectures—including ResNet, MobileNet, EfficientNet, and ViT variants—on datasets like CIFAR-100, Tiny ImageNet, and CUB-200-2011, the authors found significant benefits for coarse-grained classification. LDA improved accuracy by up to 4.5 percentage points while reducing feature dimensionality by 48-87% in most coarse-grained scenarios. However, the method consistently underperformed on fine-grained tasks such as CUB-200, where subtle cues are critical. The study concludes that plain LDA provides the optimal accuracy-cost tradeoff for coarse-grained settings, whereas complex supervised reduction methods rarely justify their additional computational costs. These findings offer practical guidance for integrating post-hoc supervised projections into frozen-feature image classification systems.
cs.AI updates on arXiv.org