Text-Guided Multi-Scale Frequency Representation Adaptation

Researchers have introduced FreqAdapter, a novel parameter-efficient fine-tuning method designed to enhance the adaptability of pre-trained multimodal models. Addressing limitations in existing techniques, such as information redundancy in signal space and fixed prompts that ignore multi-scale signal characteristics, FreqAdapter operates in the frequency domain. It integrates textual information to perform multi-scale fine-tuning and employs a strategy to optimize receptive fields across different frequency ranges. This approach significantly boosts the model's representational capacity. Extensive experiments conducted on prominent multimodal models, including CLIP and LLaVA, demonstrate that FreqAdapter improves both performance and efficiency with minimal computational cost. Notably, the method achieves fast convergence within a single training epoch. The study highlights the potential of frequency-domain adaptation for optimizing large-scale AI models. The associated code has been made publicly available to facilitate further research and application in computer vision and pattern recognition tasks.