Adaptive Computation Depth via Learned Token Routing in Transformers
A new research paper introduces Token-Selective Attention (TSA), a novel mechanism designed to enhance the efficiency of standard transformer architectures. Unlike traditional models that apply uniform computational depth to all tokens, TSA employs a learned per-token gate to dynamically adjust processing based on contextual difficulty. This gate, implemented as a lightweight two-layer multi-layer perceptron, generates continuous halting probabilities, allowing the model to skip unnecessary layers for simpler tokens. The method is end-to-end differentiable and requires only a 1.7% parameter overhead without altering the base architecture. Experimental results on character-level language modeling tasks, including Tiny-Shakespeare and enwik8 datasets, demonstrate that TSA reduces token-layer operations by 14-23% with less than 0.5% loss in quality. Furthermore, it outperforms early exit strategies by achieving lower validation loss at matched efficiency levels. The learned routing effectively transfers to inference-time sparse execution, providing tangible wall-clock speedups. This development represents a significant step toward more adaptive and resource-efficient deep learning models.
Wire timeline
Adaptive Computation Depth via Learned Token Routing in Transformers
A new research paper introduces Token-Selective Attention (TSA), a novel mechanism designed to enhance the efficiency of standard transformer architectures. Unlike traditional models that apply uniform computational depth to all tokens, TSA employs a learned per-token gate to dynamically adjust processing based on contextual difficulty. This gate, implemented as a lightweight two-layer multi-layer perceptron, generates continuous halting probabilities, allowing the model to skip unnecessary layers for simpler tokens. The method is end-to-end differentiable and requires only a 1.7% parameter overhead without altering the base architecture. Experimental results on character-level language modeling tasks, including Tiny-Shakespeare and enwik8 datasets, demonstrate that TSA reduces token-layer operations by 14-23% with less than 0.5% loss in quality. Furthermore, it outperforms early exit strategies by achieving lower validation loss at matched efficiency levels. The learned routing effectively transfers to inference-time sparse execution, providing tangible wall-clock speedups. This development represents a significant step toward more adaptive and resource-efficient deep learning models.
cs.AI updates on arXiv.org