构建可执行旅行计划的严格评测基准，检验大模型真实规划能力。
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning
- 设计多约束联合评测任务，强制验证行程可行性与逻辑一致性。
- 15个大模型仅46.2%任务能生成完整可行计划，短板在隐性需求理解。
- 全规则自动评分+人工标注真值，确保结果可复现、可审计。
旅行规划是对工具型大模型的严苛考验：一个可用行程需在多个维度同时正确——航班、酒店、景点必须真实可预订，时间空间安排合理，总预算合规，且满足部分未明说的旅客偏好。现有评测仅逐项打分，依赖模糊或大模型评分，无法验证计划可执行性，也缺乏可复现性。我们提出TREK（旅行推理与评估套件），一个面向可行行程生成的基准：要求生成单一计划，在约束正确、无幻觉、时空可执行、预算合规及响应隐性需求上全部达标。TREK包含800个任务（533个可行，267个可证明不可行，含类型化原因），基于包含212,530条记录的合成知识库（覆盖375个城市和13种人物角色），通过经验证的RESTful API沙箱提供。每个任务由完全确定性的规则引擎评分，无大模型裁判，并附有人工验证的黄金参考答案（在该评估器下得分为1.0），确保天花板可达成，所有差距均为代理能力不足所致。对15个大模型在九个约束维度上的评估显示，最强模型（GPT-5.6）仅在46.2%的可解任务中生成完全可行计划，中位数为6.6%，最低为0.0%；满足隐性需求成为普遍瓶颈，前沿模型仍未解决。我们开源数据集、工具沙箱、规则评估器及代理代码，打造可复现的基准。
原文摘要 · Abstract (English)
Travel planning is a demanding stress test for tool-using LLM agents: a usable itinerary is a single artifact that must be right along many axes at once - every flight, hotel, and attraction must exist and be bookable, the days must be physically traversable, the total must clear a budget, and the plan must serve a traveler whose needs are only partly stated. Existing agent benchmarks reward these properties one at a time and grade the final output with soft or LLM-judged rubrics, which cannot certify that a returned plan is executable and are neither reproducible nor auditable. We introduce TREK (Travel Reasoning and Evaluation Kit), a benchmark for feasible itinerary synthesis: producing a single plan that is jointly constraint-correct, hallucination-free, spatio-temporally executable, budget-valid, and responsive to the traveler's unstated persona needs. TREK comprises 800 multi-constraint tasks - 533 feasible and 267 provably infeasible with typed route/entity/budget causes - over a synthetic, internally consistent knowledge base of 212,530 records across 375 cities and 13 personas, served through a production-style tool sandbox of validated RESTful APIs. Every task is scored by a fully deterministic, rule-based evaluator with no LLM judge and ships a human-verified gold reference that scores a perfect 1.0 under that same evaluator, so the ceiling is demonstrably achievable and every remaining gap is an agent limitation rather than scorer strictness. Evaluating 15 LLM agents across nine constraint dimensions, we find that even the strongest (GPT-5.6) produces a fully-feasible plan on only 46.2% of solvable tasks, with a median of 6.6% and a floor of 0.0%; satisfying travelers' unstated needs emerges as the universal bottleneck, unsolved even at the frontier. We release the dataset, tool sandbox, deterministic evaluator, and agent code as a fully reproducible benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。