Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging

A new research paper challenges the prevailing assumption that Transformer models in automatic sleep staging primarily succeed by learning complex long-range dependencies. Instead, the authors reveal that strong local temporal continuity is a neglected but critical property of sleep sequences. The study demonstrates that a randomly initialized Transformer, requiring no training, significantly improves sleep staging performance and consistently outperforms traditional heuristic smoothing methods. By formalizing this phenomenon through a Random Attention Prior Kernel (RAPK), the researchers show that random self-attention functions as an adaptive smoother, effectively balancing global averaging with content-based similarity while preserving essential stage transitions. Using novel metrics like the Local Smoothness Influence Index (LSII) and Weighted Transition Entropy (WTE), the team provides evidence that most performance gains stem from architectural inductive bias rather than parameter learning. These findings suggest that structure-driven smoothing mechanisms are sufficient for effective sleep staging, offering a pathway to more efficient, edge-deployable healthcare systems for large-scale physiological monitoring without the computational burden of complex dependency modeling.