AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling
Researchers have introduced AgentHER, a novel framework designed to enhance the training of Large Language Model (LLM) agents by repurposing failed execution trajectories. Current training pipelines often discard these failures, despite low success rates in benchmarks like WebArena and ToolBench. AgentHER adapts Hindsight Experience Replay (HER) to natural-language agent tasks, operating on the principle that a trajectory failing one goal may successfully demonstrate an alternative achievable goal. The system employs a four-stage pipeline including failure classification, outcome extraction, LLM-guided relabeling with confidence gating, and data packaging to convert discarded data into high-quality Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) datasets. Experimental results show that AgentHER improves performance by 7.6-11.4% over success-only SFT across multiple model families, including GPT-4o, Qwen2.5, and LLaMA-3.1. It also doubles sample efficiency and outperforms existing baselines. Robustness mechanisms, such as cross-model multi-judge verification, significantly reduce label noise and achieve high human-rated precision. The method offers a cost-effective solution for improving LLM agent capabilities by maximizing the utility of existing training data.
Wire timeline
AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling
Researchers have introduced AgentHER, a novel framework designed to enhance the training of Large Language Model (LLM) agents by repurposing failed execution trajectories. Current training pipelines often discard these failures, despite low success rates in benchmarks like WebArena and ToolBench. AgentHER adapts Hindsight Experience Replay (HER) to natural-language agent tasks, operating on the principle that a trajectory failing one goal may successfully demonstrate an alternative achievable goal. The system employs a four-stage pipeline including failure classification, outcome extraction, LLM-guided relabeling with confidence gating, and data packaging to convert discarded data into high-quality Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) datasets. Experimental results show that AgentHER improves performance by 7.6-11.4% over success-only SFT across multiple model families, including GPT-4o, Qwen2.5, and LLaMA-3.1. It also doubles sample efficiency and outperforms existing baselines. Robustness mechanisms, such as cross-model multi-judge verification, significantly reduce label noise and achieve high human-rated precision. The method offers a cost-effective solution for improving LLM agent capabilities by maximizing the utility of existing training data.
cs.AI updates on arXiv.org