构建开放旅行规划基准,提升语言智能体的复杂约束求解能力
ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents
- 设计可扩展的领域语言(DSL)实现多维度约束验证
- 基于1154人真实需求数据,约束满足率达37.0%(神经模型仅3.7%)
- 适合研究复杂任务规划与神经符号系统融合的开发者
旅行规划是语言智能体的重要现实应用,兼具实际需求与严格的约束满足挑战。现有基准多采用槽位填充范式,仅支持预定义约束的合成查询,无法捕捉自然语言交互中用户需求的组合性、多样性及隐含意图。为此,我们提出ChinaTravel:1)符合多日多景点规划的实际沙盒环境;2)可扩展的领域特定语言(DSL),支持可行性、约束满足与偏好比较的多维评估;3)包含1154名参与者真实需求的开放数据集,涵盖多样且隐含的旅行意图;4)细粒度分析表明,神经符号智能体在人类查询上实现37.0%的约束满足率,较纯神经模型提升10倍,但仍面临组合泛化挑战。ChinaTravel为语言智能体在复杂真实场景中的组合约束验证提供了基础。
原文摘要 · Abstract (English)
Travel planning stands out among real-world applications of \emph{Language Agents} because it couples significant practical demand with a rigorous constraint-satisfaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, we introduce \emph{ChinaTravel}, with four key contributions: 1) a practical sandbox aligned with the multi-day, multi-POI travel planning, 2) a compositionally generalizable domain-specific language (DSL) for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison 3) an open-ended dataset that integrates diverse travel requirements and implicit intent from 1154 human participants, and 4) fine-grained analysis reveal the potential of neuro-symbolic agents in travel planning, achieving a 37.0% constraint satisfaction rate on human queries, a 10 \times improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, ChinaTravel provides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios. Project Page: https://www.lamda.nju.edu.cn/shaojj/ChinaTravel/index.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。