TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
Researchers have introduced TinySSL, a novel framework called Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), designed to enable effective self-supervised learning for microcontroller-class models with fewer than 500,000 parameters. Addressing challenges like projection head dominance and representation bottlenecks, CA-DSSL utilizes asymmetric distillation from a frozen DINO ViT-S/16 teacher and a progressive augmentation curriculum. When tested on a MobileNetV2-0.35 backbone pretrained on CIFAR-100, the model achieved 62.7% linear-probe accuracy, significantly outperforming SimCLR-Tiny by 18 percentage points and matching SEED efficiency with substantially fewer parameters. Standard methods like BYOL-Tiny failed at this scale. In Pascal VOC detection tasks, CA-DSSL demonstrated superior performance over random initialization and SEED. The deployed model occupies only 378 KB in INT8 format, making it suitable for resource-constrained devices without inference overhead. Preliminary tests suggest its advantages are most pronounced in small-data regimes, with future work aimed at scaling to larger datasets like ImageNet-1K. This development marks a significant step in applying advanced AI techniques to edge computing hardware.
Wire timeline
TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
Researchers have introduced TinySSL, a novel framework called Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), designed to enable effective self-supervised learning for microcontroller-class models with fewer than 500,000 parameters. Addressing challenges like projection head dominance and representation bottlenecks, CA-DSSL utilizes asymmetric distillation from a frozen DINO ViT-S/16 teacher and a progressive augmentation curriculum. When tested on a MobileNetV2-0.35 backbone pretrained on CIFAR-100, the model achieved 62.7% linear-probe accuracy, significantly outperforming SimCLR-Tiny by 18 percentage points and matching SEED efficiency with substantially fewer parameters. Standard methods like BYOL-Tiny failed at this scale. In Pascal VOC detection tasks, CA-DSSL demonstrated superior performance over random initialization and SEED. The deployed model occupies only 378 KB in INT8 format, making it suitable for resource-constrained devices without inference overhead. Preliminary tests suggest its advantages are most pronounced in small-data regimes, with future work aimed at scaling to larger datasets like ImageNet-1K. This development marks a significant step in applying advanced AI techniques to edge computing hardware.
cs.AI updates on arXiv.org