评测购物智能体在复杂需求下的长期决策能力，揭示其漏解隐藏需求的短板。
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
- 任务拆分三阶段：显式查询、工具受限档案、脚本化追问，模拟真实购物场景
- 7个模型最高仅57.1%准确率，隐藏意图源导致评分下降明显
- 提供可复现的评估框架，适合研究长程对话与推理的开发者
随着基于大语言模型的购物代理进入生产环境，现有评测基准无法捕捉消费者需求的渐进呈现方式：需求可能隐含在查询中、记录在用户档案里，或仅在恰当提问后才显露。仅暴露全部意图并仅评价最终选择的基准，既无法体现长期任务挑战，也无法说明代理遗漏了哪类需求。为此，我们提出 EComAgentBench，一个基于真实 Amazon 产品与评论的 662 项任务基准。每项任务将需求分散于可见查询、工具限制的档案和脚本化澄清环节；代理需挖掘隐藏意图，用属性与评论证据验证候选商品，并在 100 次工具调用内选出唯一产品。同时，带有类型与来源标签的评分标准对每项任务进行细粒度打分，将每个失败归因于具体需求及其来源。构建过程自动化且可靠：所有答案在生成文本前即以代码形式固定，每项样本均经验证。对七种模型的评估显示，即使最强模型整体准确率也仅为 57.1%，且来自隐藏源的需求满足率显著下降。我们认为，EComAgentBench 将成为推动购物代理从单次查询搜索迈向长期可信赖辅助的可复现基础。
原文摘要 · Abstract (English)
As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。