BADIT: Mitigating Cross-Task Interference in LLM Multi-Task Instruct-Tuning

Researchers have introduced a novel method called Basic Abilities Decomposition for multi-task Instruct-Tuning (BADIT) to address cross-task interference in Large Language Models (LLMs). While multi-task instruct-tuning drives LLM performance, it often suffers from conflicting gradients across shared parameters. Previous solutions like task-specific neuron selection failed to fully eliminate this interference. The study empirically reveals that co-activated parameters form base groups, suggesting LLMs encode orthogonal basic abilities. BADIT decomposes LLM parameters into orthogonal high-singular-value LoRA experts representing these abilities and enforces orthogonality via spherical clustering of rank-1 components during training. Extensive experiments on the SuperNI benchmark involving six different LLMs demonstrate that BADIT outperforms state-of-the-art methods and significantly mitigates cross-task interference. This advancement offers a more efficient training paradigm for enhancing the versatility and accuracy of large language models in handling diverse tasks simultaneously without performance degradation due to parameter conflict.