Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity

Researchers have identified a critical issue termed 'rank collapse' in Federated Low-Rank Adaptation (FedLoRA), a technique used for efficient and privacy-preserving fine-tuning of foundation models. In scenarios with heterogeneous client resources, FedLoRA often employs varying LoRA ranks. However, the study reveals that energy from global updates concentrates in the minimum shared rank, leading to suboptimal performance. The root cause is a mismatch between rank-agnostic aggregation weights and rank-dependent client contributions, which suppresses higher-rank updates geometrically over time. To address this, the authors propose raFLoRA, a rank-partitioned aggregation method. This approach decomposes local updates into rank partitions and aggregates them based on effective client contributions. Extensive experiments across vision, language, and reasoning tasks demonstrate that raFLoRA effectively prevents rank collapse, enhances model performance, and improves robustness compared to existing FedLoRA baselines. This academic paper, published on arXiv, offers a significant theoretical and practical advancement for distributed machine learning systems dealing with data and system heterogeneity.