Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs

A new research paper submitted to arXiv addresses a critical challenge in deploying Large Language Models (LLMs): the performance degradation caused by extremely low-bit quantization. While existing methods primarily focus on improving numerical accuracy to mitigate loss, this study reveals that extremely quantized LLMs suffer from systematic smoothness degradation beyond mere precision errors. The authors demonstrate that as quantization bit-width decreases, the smoothness of the model's output deteriorates significantly. Through sequence neighborhood modeling, they found that quantized models exhibit a rapid reduction in effective token candidates within the prediction neighborhood, leading to a sparser decoding tree and lower generation quality. To address this, the researchers introduced a simple smoothness-preserving principle applicable to both post-training quantization and quantization-aware training. Their experiments show that preserving smoothness yields additional performance gains beyond what is achieved by optimizing numerical accuracy alone. The paper aims to establish smoothness preservation as a key design consideration for future extreme quantization techniques, offering a new direction for efficient LLM deployment. The associated code has been made publicly available to support further research and implementation.