Sink vs. Diagonal Patterns as Mechanisms for Attention Switch and Oversmoothing Prevention

A new research paper submitted to arXiv investigates the functional roles of sink tokens and diagonal patterns within transformer architectures, specifically regarding attention switching and the prevention of oversmoothing. The authors analyze the geometric conditions required for sink representation, demonstrating a necessary alignment between sink embeddings and other token embeddings. The study refines existing theories by specifying conditions under which dense attention smooths data more than sparse attention, verifying these conditions empirically. Furthermore, the paper proves an equivalence between sinks and hard attention switches, where attention output is zero. By relaxing this constraint to allow token self-communication, the researchers provide a quantitative comparison of the computational costs associated with representing sinks versus diagonal patterns. The findings explain why sinks are preferred in pretrained transformers and clarify when attention layers function similarly to Multi-Layer Perceptrons (MLPs) when token communication is unnecessary. This work bridges the gap between theoretical requirements for oversmoothing prevention and the practical mechanisms provided by sinks in deep learning models.