构建首个电商脚本规划基准,让AI学会按步骤推荐商品
EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association
- 分三步规划购物流程,用意图匹配商品
- 生成60万条真实电商脚本,覆盖240万商品
- 揭示大模型在商品推荐上仍有明显短板
目标导向的脚本规划是人类完成日常任务的重要能力。在电商场景中,用户期望大语言模型能生成连贯的购物步骤并推荐对应商品,提升购物效率。然而,现有模型难以同时完成脚本生成与商品检索,且因动作描述与搜索词语义不一致导致匹配困难,缺乏评估标准。本文首次正式定义电商脚本规划(EcomScript)为三个连续子任务,并提出新框架,通过动作与购买意图的语义相似性关联商品,实现产品增强型脚本生成。基于真实电商数据,构建首个大规模数据集EcomScriptBench,包含605,229条脚本和240万商品。人工标注部分样本作为黄金标准,形成评估基准。大量实验表明,当前(大)模型在该任务上表现不佳,即使微调后仍存显著差距,而引入购买意图可有效提升性能。
原文摘要 · Abstract (English)
Goal-oriented script planning, or the ability to devise coherent sequences of actions toward specific goals, is commonly employed by humans to plan for typical activities. In e-commerce, customers increasingly seek LLM-based assistants to generate scripts and recommend products at each step, thereby facilitating convenient and efficient shopping experiences. However, this capability remains underexplored due to several challenges, including the inability of LLMs to simultaneously conduct script planning and product retrieval, difficulties in matching products caused by semantic discrepancies between planned actions and search queries, and a lack of methods and benchmark data for evaluation. In this paper, we step forward by formally defining the task of E-commerce Script Planning (EcomScript) as three sequential subtasks. We propose a novel framework that enables the scalable generation of product-enriched scripts by associating products with each step based on the semantic similarity between the actions and their purchase intentions. By applying our framework to real-world e-commerce data, we construct the very first large-scale EcomScript dataset, EcomScriptBench, which includes 605,229 scripts sourced from 2.4 million products. Human annotations are then conducted to provide gold labels for a sampled subset, forming an evaluation benchmark. Extensive experiments reveal that current (L)LMs face significant challenges with EcomScript tasks, even after fine-tuning, while injecting product purchase intentions improves their performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。