SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
Researchers Yipin Guo and Siddharth Joshi have introduced SplitZip, a novel GPU-friendly lossless compression technique designed to optimize Key-Value (KV) cache transfer in disaggregated Large Language Model (LLM) serving systems. As LLM architectures increasingly separate compute-bound prefill phases from memory-bound decode phases across different physical systems, data transfer bottlenecks significantly hinder performance, especially for long-input workloads. Existing codecs are often too slow or CPU-dependent for this real-time context. SplitZip addresses this by exploiting redundancy in floating-point exponents, using an offline-calibrated codebook for frequent values and a sparse escape stream for rare ones. This method ensures bitwise preservation of KV tensors while achieving high throughput rates of 613.3 GB/s for compression and 2181.8 GB/s for decompression on BF16 tensors. Experimental results demonstrate substantial improvements, including a 1.32x speedup in KV cache transfer, a 1.30x reduction in Time to First Token (TTFT), and a 1.23x increase in request throughput. The technology integrates seamlessly into existing frameworks without altering model execution, marking a significant advancement in efficient, scalable LLM infrastructure.
Wire timeline
SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
Researchers Yipin Guo and Siddharth Joshi have introduced SplitZip, a novel GPU-friendly lossless compression technique designed to optimize Key-Value (KV) cache transfer in disaggregated Large Language Model (LLM) serving systems. As LLM architectures increasingly separate compute-bound prefill phases from memory-bound decode phases across different physical systems, data transfer bottlenecks significantly hinder performance, especially for long-input workloads. Existing codecs are often too slow or CPU-dependent for this real-time context. SplitZip addresses this by exploiting redundancy in floating-point exponents, using an offline-calibrated codebook for frequent values and a sparse escape stream for rare ones. This method ensures bitwise preservation of KV tensors while achieving high throughput rates of 613.3 GB/s for compression and 2181.8 GB/s for decompression on BF16 tensors. Experimental results demonstrate substantial improvements, including a 1.32x speedup in KV cache transfer, a 1.30x reduction in Time to First Token (TTFT), and a 1.23x increase in request throughput. The technology integrates seamlessly into existing frameworks without altering model execution, marking a significant advancement in efficient, scalable LLM infrastructure.
cs.AI updates on arXiv.org