AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems
Researchers have introduced AgentForesight, a novel framework designed to predict and prevent failures in LLM-based multi-agent systems during long-horizon tasks. Unlike existing post-hoc methods that diagnose errors after task completion, AgentForesight employs online auditing to identify decisive errors in real-time, allowing for immediate intervention. The team curated AFTraj-2K, a comprehensive corpus of agentic trajectories across Coding, Math, and Agentic domains, annotated with precise error steps via multi-LLM consensus. Using this dataset, they developed AgentForesight-7B, a compact auditor trained with a coarse-to-fine reinforcement learning strategy. This model first establishes a risk-anticipation prior and then refines it to localize errors accurately. Benchmarks show that AgentForesight-7B outperforms leading proprietary models like GPT-4.1 and DeepSeek-V4-Pro, achieving a 19.9% performance gain and three times lower step localization error. This advancement shifts the paradigm from retrospective failure analysis to proactive deployment-time intervention, significantly enhancing the reliability of autonomous multi-agent systems.
Wire timeline
AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems
Researchers have introduced AgentForesight, a novel framework designed to predict and prevent failures in LLM-based multi-agent systems during long-horizon tasks. Unlike existing post-hoc methods that diagnose errors after task completion, AgentForesight employs online auditing to identify decisive errors in real-time, allowing for immediate intervention. The team curated AFTraj-2K, a comprehensive corpus of agentic trajectories across Coding, Math, and Agentic domains, annotated with precise error steps via multi-LLM consensus. Using this dataset, they developed AgentForesight-7B, a compact auditor trained with a coarse-to-fine reinforcement learning strategy. This model first establishes a risk-anticipation prior and then refines it to localize errors accurately. Benchmarks show that AgentForesight-7B outperforms leading proprietary models like GPT-4.1 and DeepSeek-V4-Pro, achieving a 19.9% performance gain and three times lower step localization error. This advancement shifts the paradigm from retrospective failure analysis to proactive deployment-time intervention, significantly enhancing the reliability of autonomous multi-agent systems.
cs.AI updates on arXiv.org