Selective LoRA for Visual Tokens and Attention Heads
Researchers have introduced Image-LoRA, a novel parameter-efficient fine-tuning (PEFT) method designed specifically for Vision Language Models (VLMs). Unlike standard Low-rank adaptation (LoRA), which applies updates uniformly across all tokens and attention heads, Image-LoRA selectively targets only visual tokens and a compact subset of attention heads within the value path. This selection is driven by a one-pass influence estimate using a rank-1 visual-token-only probe. The approach significantly reduces trainable parameters and computational costs (FLOPs) while preserving the pure-text forward pass of the frozen backbone when no visual inputs are present. Experimental results demonstrate that Image-LoRA matches or closely approaches the performance of standard LoRA on visual localization benchmarks, particularly excelling in regimes with high image-to-text token ratios. The method’s effectiveness is further validated on TextVQA, VideoQA, and GSM8K datasets, showing that its selective information bottleneck can yield performance gains over traditional methods. This development addresses the heterogeneous nature of VLM inputs, offering a more efficient adaptation strategy for multimodal AI systems.
Wire timeline
Selective LoRA for Visual Tokens and Attention Heads
Researchers have introduced Image-LoRA, a novel parameter-efficient fine-tuning (PEFT) method designed specifically for Vision Language Models (VLMs). Unlike standard Low-rank adaptation (LoRA), which applies updates uniformly across all tokens and attention heads, Image-LoRA selectively targets only visual tokens and a compact subset of attention heads within the value path. This selection is driven by a one-pass influence estimate using a rank-1 visual-token-only probe. The approach significantly reduces trainable parameters and computational costs (FLOPs) while preserving the pure-text forward pass of the frozen backbone when no visual inputs are present. Experimental results demonstrate that Image-LoRA matches or closely approaches the performance of standard LoRA on visual localization benchmarks, particularly excelling in regimes with high image-to-text token ratios. The method’s effectiveness is further validated on TextVQA, VideoQA, and GSM8K datasets, showing that its selective information bottleneck can yield performance gains over traditional methods. This development addresses the heterogeneous nature of VLM inputs, offering a more efficient adaptation strategy for multimodal AI systems.
cs.AI updates on arXiv.org