Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
A new research paper titled 'Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers' introduces a method to address the limitation of transformers exposing only one inference cost per trained model. The study proposes a monotone head-gating mechanism conditioned on a requested attention budget, allowing deployed systems to operate at multiple cost-quality points. Dense warm-starting is highlighted as crucial for stability. Experimental results on synthetic tasks show high accuracy at varying attention costs. On the AG News dataset, hard-gate adaptation achieved an 82.1% accuracy with a 1.28x speedup on single-thread CPU at a 0.50 budget. Similarly, budgeted structural pruning in pretrained BERT-Mini models reached 87.6% accuracy with a 1.20x speedup. On DBpedia14, the method outperformed dense full attention. The authors emphasize that this work is a reproducible feasibility study demonstrating how a single controllable checkpoint can trade attention cost for accuracy, converting soft cost control into measured structural speedups on small CPU benchmarks, rather than claiming universal dominance over existing methods.
Wire timeline
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
A new research paper titled 'Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers' introduces a method to address the limitation of transformers exposing only one inference cost per trained model. The study proposes a monotone head-gating mechanism conditioned on a requested attention budget, allowing deployed systems to operate at multiple cost-quality points. Dense warm-starting is highlighted as crucial for stability. Experimental results on synthetic tasks show high accuracy at varying attention costs. On the AG News dataset, hard-gate adaptation achieved an 82.1% accuracy with a 1.28x speedup on single-thread CPU at a 0.50 budget. Similarly, budgeted structural pruning in pretrained BERT-Mini models reached 87.6% accuracy with a 1.20x speedup. On DBpedia14, the method outperformed dense full attention. The authors emphasize that this work is a reproducible feasibility study demonstrating how a single controllable checkpoint can trade attention cost for accuracy, converting soft cost control into measured structural speedups on small CPU benchmarks, rather than claiming universal dominance over existing methods.
cs.AI updates on arXiv.org