Anchored Bipolicy Self-Play Enhances AI Safety by Breaking Self-Consistency

Researchers have introduced Anchored Bipolicy Self-Play, a novel method to improve AI safety by addressing limitations in traditional self-play red teaming. Standard self-play, where identical model instances act as attacker and defender, often collapses into self-consistency, failing to generate sufficient adversarial pressure and leading to trivial Nash equilibria like always-refusing strategies. To overcome this, the new approach trains distinct role-specific LoRA adapters on a frozen base model, ensuring explicit role separation while maintaining optimization stability. Evaluations on Qwen2.5 models (3B, 7B, 14B) demonstrate that this method achieves up to 100x greater parameter efficiency than fine-tuning and offers consistent safety improvements. The resulting models show enhanced robustness against jailbreaks without compromising reasoning capabilities. Cross-play experiments confirm that these specialized attacker and defender models outperform standard self-play configurations in adversarial defense. This development marks a significant step in creating more robust and efficient safety mechanisms for large language models, highlighting the importance of architectural separation in adversarial training dynamics.