Key-Value Means: A Novel Block-Recurrence for Efficient Attention
Researchers Daniel Goldstein and Eugene Cheah have introduced Key-Value Means (KVM), a new block-recurrence mechanism for attention models that supports both fixed-size and growing state configurations. Published on arXiv, this innovation allows strong transformer baselines to function as efficient O(N) chunked RNNs with minimal additional parameters. The study demonstrates that transformers equipped with growable KVM caches achieve competitive performance in long-context tests, featuring subquadratic prefill time and sublinear state growth. KVM is designed for ease of implementation using standard operations without custom kernels, supporting chunk-wise parallelizable training and prefill. It effectively combines the benefits of traditional transformers, such as expandable context memory, with the efficiency of linear RNNs. The method can be applied to every layer to save KV-cache memory or used in hybrid solutions with LRNN layers. The authors have released their code and trained models under the Apache 2.0 license, aiming to provide a unified package that offers flexible choices for prefill time complexity between O(N) and O(N^2).
Wire timeline
Key-Value Means: A Novel Block-Recurrence for Efficient Attention
Researchers Daniel Goldstein and Eugene Cheah have introduced Key-Value Means (KVM), a new block-recurrence mechanism for attention models that supports both fixed-size and growing state configurations. Published on arXiv, this innovation allows strong transformer baselines to function as efficient O(N) chunked RNNs with minimal additional parameters. The study demonstrates that transformers equipped with growable KVM caches achieve competitive performance in long-context tests, featuring subquadratic prefill time and sublinear state growth. KVM is designed for ease of implementation using standard operations without custom kernels, supporting chunk-wise parallelizable training and prefill. It effectively combines the benefits of traditional transformers, such as expandable context memory, with the efficiency of linear RNNs. The method can be applied to every layer to save KV-cache memory or used in hybrid solutions with LRNN layers. The authors have released their code and trained models under the Apache 2.0 license, aiming to provide a unified package that offers flexible choices for prefill time complexity between O(N) and O(N^2).
cs.AI updates on arXiv.org