arXiv:2607.28956cs.AI2026-07被引 1

评测大模型在电商运营中长期连贯决策能力,发现顶尖模型仅达人类27.3%水平。

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

  • 构建365天电商仿真环境,模拟采购、定价、现金流等长期任务
  • 8个大模型在48次运行中平均净资产仅为人类的27.3%
  • 适合评估智能体在延迟反馈下的持续决策能力

大语言模型智能体在现实应用中常需保持长期连贯性,即在长时间跨度内维持目标导向行为并根据累积证据调整决策。现有评测多聚焦短期任务,缺乏对长期适应性的评估机制。本文提出MerchantBench,一个基于98,843条真实电商产品记录的365天订单级仿真平台,配备26种工具供智能体交互。该平台将即时上游供应商事件与延迟下游订单结果耦合,要求智能体追踪每个订单生命周期并回溯早期决策。在两种智能体框架下,对8个大模型进行48次365天的模拟测试,结果显示即使最先进的模型配置也仅达到人类参与者平均净资产的27.3%,揭示了当前大模型在长期协同决策上的显著差距。

原文摘要 · Abstract (English)

Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.

大模型智能体长期决策电商仿真评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。