AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
Researchers have introduced AdaPreLoRA, a novel optimization method for Low-Rank Adaptation (LoRA) in large language models. Standard LoRA techniques face challenges because the Jacobian of the generator mapping factors to weight matrices is rank-deficient, making it difficult to uniquely invert preconditioned directions back to factor-space updates. AdaPreLoRA addresses this by adopting the Adafactor diagonal Kronecker preconditioner on the weight space and selecting a factor-space solution that minimizes imbalance between factor contributions. This approach ensures the update is the closest LoRA approximation to the preconditioned direction under a weighted norm. The method operates with efficient memory usage at O((m+n)r). Empirical evaluations across GPT-2, Mistral-7B, and Qwen2-7B models, as well as diffusion-model personalization tasks, demonstrate that AdaPreLoRA is competitive with or superior to existing LoRA optimizers while maintaining low peak GPU memory requirements. This development offers a unified framework for understanding LoRA optimizers and provides a practical, high-performance solution for fine-tuning large-scale AI models.
Wire timeline
AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
Researchers have introduced AdaPreLoRA, a novel optimization method for Low-Rank Adaptation (LoRA) in large language models. Standard LoRA techniques face challenges because the Jacobian of the generator mapping factors to weight matrices is rank-deficient, making it difficult to uniquely invert preconditioned directions back to factor-space updates. AdaPreLoRA addresses this by adopting the Adafactor diagonal Kronecker preconditioner on the weight space and selecting a factor-space solution that minimizes imbalance between factor contributions. This approach ensures the update is the closest LoRA approximation to the preconditioned direction under a weighted norm. The method operates with efficient memory usage at O((m+n)r). Empirical evaluations across GPT-2, Mistral-7B, and Qwen2-7B models, as well as diffusion-model personalization tasks, demonstrate that AdaPreLoRA is competitive with or superior to existing LoRA optimizers while maintaining low peak GPU memory requirements. This development offers a unified framework for understanding LoRA optimizers and provides a practical, high-performance solution for fine-tuning large-scale AI models.
cs.AI updates on arXiv.org