RDKV: Joint Eviction and Quantization for Efficient KV Cache Compression in LLMs
Researchers have introduced RDKV, a novel method for optimizing Key-Value (KV) cache compression in Large Language Models (LLMs) to address memory bottlenecks during long-context inference. Traditional approaches typically handle cache eviction and quantization separately, but RDKV treats them as interconnected components of a unified rate-distortion problem. By calculating the distortion impact on attention computations, the algorithm assigns optimal bit-widths to tokens or channels, ranging from full precision to zero bits, using a reverse water-filling strategy after the prefilling stage. Experimental results on benchmarks such as LongBench, RULER, and InfiniteBench demonstrate that RDKV outperforms existing baselines by an average of 9.1%. Notably, it retains 97.81% of full-cache accuracy while using only 2.48% of the cache. Furthermore, with a 128K context length, the method achieves a 4.5x decoding speedup and reduces peak memory usage by 1.9x compared to FlashAttention-2, offering significant efficiency improvements without compromising model performance.
Wire timeline
RDKV: Joint Eviction and Quantization for Efficient KV Cache Compression in LLMs
Researchers have introduced RDKV, a novel method for optimizing Key-Value (KV) cache compression in Large Language Models (LLMs) to address memory bottlenecks during long-context inference. Traditional approaches typically handle cache eviction and quantization separately, but RDKV treats them as interconnected components of a unified rate-distortion problem. By calculating the distortion impact on attention computations, the algorithm assigns optimal bit-widths to tokens or channels, ranging from full precision to zero bits, using a reverse water-filling strategy after the prefilling stage. Experimental results on benchmarks such as LongBench, RULER, and InfiniteBench demonstrate that RDKV outperforms existing baselines by an average of 9.1%. Notably, it retains 97.81% of full-cache accuracy while using only 2.48% of the cache. Furthermore, with a 128K context length, the method achieves a 4.5x decoding speedup and reduces peak memory usage by 1.9x compared to FlashAttention-2, offering significant efficiency improvements without compromising model performance.
cs.AI updates on arXiv.org