测试大模型在动态规划中的灵活应变能力,发现现有评估方法有局限。
Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents
- 设计多轮动态约束引入与优先级冲突场景,模拟真实规划过程。
- 模型在多轮调整中表现远低于单轮任务,且易忽略高优先级约束。
- 适合研究智能代理、复杂决策系统或评估大模型实际规划能力的人看。
现实世界的规划问题需要不断适应变化的需求并平衡相互竞争的约束条件。然而,当前评估大语言模型(LLM)规划能力的基准主要集中在静态、单轮场景。我们提出 Flex-TravelPlanner,一个用于评估语言模型在动态规划场景中灵活推理能力的基准。基于 TravelPlanner 数据集,我们引入两种新评估设置:(1) 多轮逐步引入约束;(2) 显式设定优先级的竞争性约束场景。对 GPT-4o 与 Llama 3.1 70B 的分析显示:单轮任务表现无法预测模型在多轮调整中的适应能力;约束引入顺序显著影响结果;模型在优先级处理上表现不佳,常错误地优先考虑新引入的低优先级偏好,而非已有高优先级约束。这些发现凸显了在更贴近现实的动态场景中评估模型的重要性,并为提升复杂规划任务性能指明方向。代码与数据集已公开于 https://github.com/juhyunohh/FlexTravelBench。
原文摘要 · Abstract (English)
Real-world planning problems require constant adaptation to changing requirements and balancing of competing constraints. However, current benchmarks for evaluating LLMs' planning capabilities primarily focus on static, single-turn scenarios. We introduce Flex-TravelPlanner, a benchmark that evaluates language models' ability to reason flexibly in dynamic planning scenarios. Building on the TravelPlanner dataset~\citep{xie2024travelplanner}, we introduce two novel evaluation settings: (1) sequential constraint introduction across multiple turns, and (2) scenarios with explicitly prioritized competing constraints. Our analysis of GPT-4o and Llama 3.1 70B reveals several key findings: models' performance on single-turn tasks poorly predicts their ability to adapt plans across multiple turns; constraint introduction order significantly affects performance; and models struggle with constraint prioritization, often incorrectly favoring newly introduced lower priority preferences over existing higher-priority constraints. These findings highlight the importance of evaluating LLMs in more realistic, dynamic planning scenarios and suggest specific directions for improving model performance on complex planning tasks. The code and dataset for our framework are publicly available at https://github.com/juhyunohh/FlexTravelBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。