VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models

Researchers have introduced VC-Soup, a novel framework designed to improve the alignment of Large Language Models (LLMs) with multiple, potentially conflicting human values. As LLMs increasingly influence decision-making and content generation, ensuring they adhere to diverse ethical standards is critical for trustworthy AI. Existing methods, such as reward reweighting and model merging, often struggle with high computational costs and performance degradation due to value conflicts. VC-Soup addresses these issues by focusing on value consistency within training data. The framework employs a new metric based on cosine similarity to filter out low-consistency preference pairs, thereby creating smoother policy models that preserve linear mode connectivity. These policies are then linearly combined and refined using Pareto filtering to achieve balanced multi-value performance. Extensive experiments and theoretical analysis demonstrate that VC-Soup effectively mitigates value conflicts and consistently outperforms current multi-value alignment techniques, offering a more efficient and robust solution for aligning AI systems with complex human values.