REI-Bench: Evaluating Embodied Agents on Vague Human Instructions in Task Planning
Researchers have introduced REI-Bench, the first robot task planning benchmark designed to systematically model vague referring expressions (REs) in human instructions. While large language model (LLM)-based planners perform well with clear commands, they struggle with the ambiguity common in real-world interactions, particularly among non-expert users like the elderly and children. The study reveals that such vagueness can degrade robot planning success rates by up to 36.9%, primarily due to missing objects in the planning process. To address this, the authors propose a method called task-oriented context cognition, which generates clearer instructions for robots. This approach achieves state-of-the-art performance, outperforming existing techniques like aware prompts, chains of thought, and in-context learning. By tackling the issue of linguistic vagueness grounded in pragmatic theory, this work aims to enhance the accessibility and reliability of embodied agents in domestic and service environments, making them more effective for diverse user groups.
Wire timeline
REI-Bench: Evaluating Embodied Agents on Vague Human Instructions in Task Planning
Researchers have introduced REI-Bench, the first robot task planning benchmark designed to systematically model vague referring expressions (REs) in human instructions. While large language model (LLM)-based planners perform well with clear commands, they struggle with the ambiguity common in real-world interactions, particularly among non-expert users like the elderly and children. The study reveals that such vagueness can degrade robot planning success rates by up to 36.9%, primarily due to missing objects in the planning process. To address this, the authors propose a method called task-oriented context cognition, which generates clearer instructions for robots. This approach achieves state-of-the-art performance, outperforming existing techniques like aware prompts, chains of thought, and in-context learning. By tackling the issue of linguistic vagueness grounded in pragmatic theory, this work aims to enhance the accessibility and reliability of embodied agents in domestic and service environments, making them more effective for diverse user groups.
cs.AI updates on arXiv.org