测试大模型智能体在异步任务中的长期规划能力，发现现有方法表现严重不足。
Robotouille: An Asynchronous Planning Benchmark for LLM Agents
- 构建异步规划基准环境Robotouille，模拟复杂并发任务场景。
- GPT-4o的ReAct方法在同步任务上准确率达47%，异步任务仅11%。
- 强调需引入长周期反馈与自我检查机制以提升智能体执行能力。
高效的异步规划——即在存在时间延迟、需处理多样长周期任务或与其他智能体协作时，对并行或串行状态与动作进行高效推理和规划——是智能体的关键能力。尽管大语言模型（LLM）智能体在高层任务规划中展现出潜力，但现有基准主要聚焦短周期任务，未评估其异步规划能力。我们提出Robotouille，一个挑战性基准环境，用于测试LLM智能体处理长周期异步场景的能力。我们的同步与异步数据集逐步增加规划复杂度，超越现有基准，要求智能体管理重叠任务与中断。结果显示，ReAct（gpt4-o）在同步任务上达到47%准确率，但在异步任务上仅11%，表明仍有巨大提升空间。我们进一步分析失败模式，揭示需要更好地融入长周期反馈，并在任务执行中自检推理过程。代码已开源：https://github.com/portal-cornell/robotouille。
原文摘要 · Abstract (English)
Effective asynchronous planning, or the ability to efficiently reason and plan over states and actions that must happen in parallel or sequentially, is essential for agents that must account for time delays, reason over diverse long-horizon tasks, and collaborate with other agents. While large language model (LLM) agents show promise in high-level task planning, current benchmarks focus primarily on short-horizon tasks and do not evaluate such asynchronous planning capabilities. We introduce Robotouille, a challenging benchmark environment designed to test LLM agents' ability to handle long-horizon asynchronous scenarios. Our synchronous and asynchronous datasets capture increasingly complex planning challenges that go beyond existing benchmarks, requiring agents to manage overlapping tasks and interruptions. Our results show that ReAct (gpt4-o) achieves 47% on synchronous tasks but only 11% on asynchronous tasks, highlighting significant room for improvement. We further analyze failure modes, demonstrating the need for LLM agents to better incorporate long-horizon feedback and self-audit their reasoning during task execution. Code is available at https://github.com/portal-cornell/robotouille.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。