AIPO: Enhancing LLM Reasoning via Active Multi-Agent Interaction
Researchers have introduced AIPO, a novel reinforcement learning framework designed to enhance the reasoning capabilities of Large Language Models (LLMs). Addressing the limitations of existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, which are often constrained by the policy model's inherent boundaries, AIPO leverages active multi-agent interaction during the exploration phase. The framework enables the policy model to proactively consult three specialized collaborative agents—a Verify Agent, a Knowledge Agent, and a Reasoning Agent—when encountering reasoning bottlenecks. This approach provides fine-grained, targeted guidance, allowing the model to actively expand its capability boundary more efficiently than methods relying on static, trajectory-level expert demonstrations. To ensure stable training, the authors implemented a tailored importance sampling coefficient and clipping strategy to mitigate off-policy bias and gradient vanishing issues. Extensive experiments on benchmarks such as AIME, MATH500, GPQA-Diamond, and LiveCodeBench demonstrate that AIPO consistently improves reasoning performance and generalizes robustly across different models. Notably, the trained policy model operates independently without requiring collaborative agents during inference.
Wire timeline
AIPO: Enhancing LLM Reasoning via Active Multi-Agent Interaction
Researchers have introduced AIPO, a novel reinforcement learning framework designed to enhance the reasoning capabilities of Large Language Models (LLMs). Addressing the limitations of existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, which are often constrained by the policy model's inherent boundaries, AIPO leverages active multi-agent interaction during the exploration phase. The framework enables the policy model to proactively consult three specialized collaborative agents—a Verify Agent, a Knowledge Agent, and a Reasoning Agent—when encountering reasoning bottlenecks. This approach provides fine-grained, targeted guidance, allowing the model to actively expand its capability boundary more efficiently than methods relying on static, trajectory-level expert demonstrations. To ensure stable training, the authors implemented a tailored importance sampling coefficient and clipping strategy to mitigate off-policy bias and gradient vanishing issues. Extensive experiments on benchmarks such as AIME, MATH500, GPQA-Diamond, and LiveCodeBench demonstrate that AIPO consistently improves reasoning performance and generalizes robustly across different models. Notably, the trained policy model operates independently without requiring collaborative agents during inference.
cs.AI updates on arXiv.org