Kaczmarz Linear Attention: A New Efficient Model for Long-Context Language Modeling
Researchers have introduced Kaczmarz Linear Attention (KLA), a novel approach to long-context language modeling that addresses the computational bottlenecks of traditional Transformer attention. Published on arXiv, this study proposes a modification to the Gated DeltaNet (GDN) model by deriving a key-norm-normalized dynamic step size inspired by the Kaczmarz projection method. Unlike previous models that rely on empirically learned coefficients, KLA uses a theoretically derived update rule to balance forgetting and information retention more effectively. Evaluations at the 0.4B parameter scale demonstrate that KLA achieves superior performance, recording a lower validation perplexity of 8.09 compared to GDN's 8.50. The model exhibits robust stability with context lengths up to 65,000 tokens and delivers significant improvements in specific tasks, including 100% accuracy in single-needle retrieval and a 7.03-point increase in multi-query associative recall. Additionally, KLA offers 2.1x higher decode throughput at 32K context length. This innovation suggests that the Kaczmarz coefficient is a critical design element for enhancing accuracy, extrapolation, and efficiency in linear-time sequence models without altering hardware kernels.
Wire timeline
Kaczmarz Linear Attention: A New Efficient Model for Long-Context Language Modeling
Researchers have introduced Kaczmarz Linear Attention (KLA), a novel approach to long-context language modeling that addresses the computational bottlenecks of traditional Transformer attention. Published on arXiv, this study proposes a modification to the Gated DeltaNet (GDN) model by deriving a key-norm-normalized dynamic step size inspired by the Kaczmarz projection method. Unlike previous models that rely on empirically learned coefficients, KLA uses a theoretically derived update rule to balance forgetting and information retention more effectively. Evaluations at the 0.4B parameter scale demonstrate that KLA achieves superior performance, recording a lower validation perplexity of 8.09 compared to GDN's 8.50. The model exhibits robust stability with context lengths up to 65,000 tokens and delivers significant improvements in specific tasks, including 100% accuracy in single-needle retrieval and a 7.03-point increase in multi-query associative recall. Additionally, KLA offers 2.1x higher decode throughput at 32K context length. This innovation suggests that the Kaczmarz coefficient is a critical design element for enhancing accuracy, extrapolation, and efficiency in linear-time sequence models without altering hardware kernels.
cs.AI updates on arXiv.org