CERSA: A Novel Memory-Efficient Fine-Tuning Method for Large Models
Researchers have introduced Cumulative Energy-Retaining Subspace Adaptation (CERSA), a new parameter-efficient fine-tuning (PEFT) technique designed to address memory constraints in large pre-trained models. While existing methods like LoRA utilize low-rank updates, they often fail to capture the full rank characteristics of weight modifications and still require significant memory for storing frozen weights. CERSA leverages singular value decomposition (SVD) to retain only principal components responsible for 90% to 95% of spectral energy, allowing for fine-tuning within this principal subspace. This approach significantly reduces memory consumption while maintaining high performance. Extensive evaluations across various domains, including image recognition, text-to-image generation, and natural language understanding, demonstrate that CERSA consistently outperforms state-of-the-art PEFT methods with substantially lower memory requirements. The research team, comprising authors from academic institutions, plans to publicly release the code to facilitate broader adoption and further development in resource-constrained AI settings.
Wire timeline
CERSA: A Novel Memory-Efficient Fine-Tuning Method for Large Models
Researchers have introduced Cumulative Energy-Retaining Subspace Adaptation (CERSA), a new parameter-efficient fine-tuning (PEFT) technique designed to address memory constraints in large pre-trained models. While existing methods like LoRA utilize low-rank updates, they often fail to capture the full rank characteristics of weight modifications and still require significant memory for storing frozen weights. CERSA leverages singular value decomposition (SVD) to retain only principal components responsible for 90% to 95% of spectral energy, allowing for fine-tuning within this principal subspace. This approach significantly reduces memory consumption while maintaining high performance. Extensive evaluations across various domains, including image recognition, text-to-image generation, and natural language understanding, demonstrate that CERSA consistently outperforms state-of-the-art PEFT methods with substantially lower memory requirements. The research team, comprising authors from academic institutions, plans to publicly release the code to facilitate broader adoption and further development in resource-constrained AI settings.
cs.AI updates on arXiv.org