测试大模型在真实购物中的表现，发现其能力远未达标。
ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?
- 构建包含145个实例的购物代理评测基准
- 顶尖模型在复杂购物任务中准确率仅15%~18%
- 适合研究AI购物助手安全与可靠性的学者
我们提出ShoppingComp，一个面向真实场景的挑战性基准，用于全面评估大语言模型驱动的购物代理在精准商品检索、专家级报告生成和安全关键决策三个核心能力上的表现。不同于以往电商评测，ShoppingComp引入大量带多重约束的商品发现查询，确保开放世界产品存在，并支持对代理输出结果的便捷验证。该基准包含145个实例、558个场景，由35位专家精心设计以反映真实的购物需求。实验结果揭示当前大模型存在显著局限：即使最先进的模型（如GPT-5.2、Gemini-3-Pro）在任务中准确率也极低（分别为17.76%和15.82%）。错误分析表明，模型在开放环境中的信息定位、多约束条件的可靠验证、噪声与矛盾证据的一致推理以及风险感知决策等方面存在根本缺陷。ShoppingComp通过暴露这些能力鸿沟，明确了人工智能系统在可被主动信任前需跨越的可信阈值。代码与数据集已开源：https://github.com/ByteDance-BandAI/ShoppingComp。
原文摘要 · Abstract (English)
We present ShoppingComp, a challenging real-world benchmark for comprehensively evaluating LLM-powered shopping agents on three core capabilities: precise product retrieval, expert-level report generation, and safety critical decision making. Unlike prior e-commerce benchmarks, ShoppingComp introduces difficult product discovery queries with many constraints, while guaranteeing open-world products and enabling easy verification of agent outputs. The benchmark comprises 145 instances and 558 scenarios, curated by 35 experts to reflect authentic shopping needs. Results reveal stark limitations of current LLMs: even state-of-the-art models achieve low performance (e.g., 17.76\% for GPT-5.2, 15.82\% for Gemini-3-Pro).Error analysis reflects limitations in core agent competencies, including information grounding in open-world environments, reliable verification of multi-constraint requirements, consistent reasoning over noisy and conflicting evidence, and risk-aware decision making. By exposing these capability gaps, ShoppingComp characterizes the trust threshold that AI systems must cross before they can be proactively trusted for reliable real-world decision making. Our code and dataset are available at https://github.com/ByteDance-BandAI/ShoppingComp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。