When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
Researchers from the machine learning community have introduced a new diagnostic framework to address bottlenecks in long-context Large Language Model (LLM) inference. The primary challenge lies in the high memory and bandwidth costs associated with reading large Key-Value (KV) caches during decoding. While KV compression aims to reduce these costs by retaining only critical cache parts, existing methods often lack clarity on why specific selectors succeed or fail. This study identifies three potential failure points: missing necessary evidence, prioritizing irrelevant tokens, or disrupting related evidence structures. To resolve this, the authors propose a fixed-contract diagnostic that isolates decision slots to evaluate selector performance. Their probe combines attention mass with estimated output changes to rank value effectively. Experimental results on LongBench, involving three models and two budgets, demonstrate that the probe yields positive outcomes in 72.6% of positive-margin cells. Additional tests using NeedleBench and RULER benchmarks support the method's efficacy in recovering decode-side evidence and preserving coupled evidence. The findings suggest a strategic order for cache compression: recover evidence, rank its output value, and preserve structural integrity during projection, offering significant insights for optimizing LLM efficiency.
Wire timeline
When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
Researchers from the machine learning community have introduced a new diagnostic framework to address bottlenecks in long-context Large Language Model (LLM) inference. The primary challenge lies in the high memory and bandwidth costs associated with reading large Key-Value (KV) caches during decoding. While KV compression aims to reduce these costs by retaining only critical cache parts, existing methods often lack clarity on why specific selectors succeed or fail. This study identifies three potential failure points: missing necessary evidence, prioritizing irrelevant tokens, or disrupting related evidence structures. To resolve this, the authors propose a fixed-contract diagnostic that isolates decision slots to evaluate selector performance. Their probe combines attention mass with estimated output changes to rank value effectively. Experimental results on LongBench, involving three models and two budgets, demonstrate that the probe yields positive outcomes in 72.6% of positive-margin cells. Additional tests using NeedleBench and RULER benchmarks support the method's efficacy in recovering decode-side evidence and preserving coupled evidence. The findings suggest a strategic order for cache compression: recover evidence, rank its output value, and preserve structural integrity during projection, offering significant insights for optimizing LLM efficiency.
cs.AI updates on arXiv.org