测试大模型在逐步暴露的双约束下能否动态调整计划。
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints

- 构建动态交互基准,逐步揭示世界与用户约束。
- 顶级模型仅达67.75%准确率,约束越多性能越差。
- 适合研究智能体自适应规划与真实场景交互的学者。
语言模型在解决现实问题时通常需同时应对世界约束和用户约束,这些约束往往未在初始阶段完全明确,而是通过交互逐步揭示。然而,现有评测基准仍未能充分探索在逐步暴露的双重约束下的自适应规划能力。为此,我们提出AdaPlanBench,一个基于307个家庭任务的动态交互基准,用于评估大语言模型(LLM)代理在逐步揭示的世界与用户约束下是否具备自适应规划与重规划能力。该基准采用可扩展的约束生成流程,为每个任务添加双重约束。运行时,代理以多轮交互方式与环境互动,隐藏约束仅在代理提出违规计划时才被揭示,要求其在累积反馈中迭代修正计划。这使得规划极具挑战性,因代理需从反馈中推断并追踪约束,同时有效重规划。对十款领先LLM的实验表明,在双重约束下的自适应规划仍具挑战性,最优模型准确率仅为67.75%。我们进一步发现,随着约束数量增加,性能持续下降,其中用户约束构成更大挑战,失败多源于物理认知薄弱与策略有效性不足。这些结果确立了AdaPlanBench作为双约束交互规划的测试平台,并凸显了大模型代理在动态揭示约束下可靠适应的难题。
原文摘要 · Abstract (English)
Planning for real-world problems by language models often involves both world and user constraints, which may not be fully specified upfront and are progressively disclosed through interaction. However, existing benchmarks still underexplore adaptive planning under such progressively revealed dual constraints. To address this gap, we introduce AdaPlanBench, a dynamic interactive benchmark for evaluating whether Large Language Model (LLM) agents can adaptively plan and re-plan under progressively revealed world and user constraints. AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task with dual constraints. At runtime, agents interact with the environment in a multi-turn protocol where hidden constraints are revealed only when the agent proposes a plan that violates them, requiring iterative plan revision under accumulating feedback. This makes planning challenging, as agents must infer and track constraints from feedback while re-planning effectively. Experiments on ten leading LLMs show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy. We further observe that performance degrades as more constraints accumulate, with user constraints posing a particularly large challenge and failures often stemming from weaker physical grounding and reduced effectiveness. These results establish AdaPlanBench as a testbed for dual-constrained interactive planning and highlight the challenge of reliable adaptation to dynamically revealed constraints in LLM agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。