构建首个覆盖57项任务的电商通用能力评测集，评估大模型做购物助手的真实水平。
Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models
- 基于真实亚马逊数据设计跨任务评测体系，涵盖四大购物核心能力。
- 测试超20个大模型，发现其在用户行为理解与隐含知识推理上普遍不足。
- 适合研究电商AI、通用对话系统及多任务学习的开发者使用。
在线购物是复杂多任务、少样本学习问题，涉及广泛且不断演变的实体、关系与任务。现有模型和评测通常针对特定任务，难以全面反映在线购物的复杂性。大型语言模型（LLMs）具备多任务与少样本学习能力，有望通过减少专用工程投入并提供交互式对话，彻底改变在线购物体验。然而，LLMs在电商场景中面临领域特有概念、隐含知识和异构用户行为等独特挑战。为此，我们提出 Shopping MMLU——一个源自真实亚马逊数据的多样化多任务电商评测基准。该基准包含57项任务，覆盖四大核心购物技能：概念理解、知识推理、用户行为对齐和多语言支持，可全面评估LLM作为通用购物助手的能力。借助 Shopping MMLU，我们对超过20个现有LLM进行了评测，揭示了构建多功能购物助手的实践洞见与未来前景。Shopping MMLU 已公开发布于 https://github.com/KL4805/ShoppingMMLU。此外，我们还在KDD Cup 2024中举办了相关竞赛，吸引超过500支团队参与，优胜方案与研讨会资料可在 https://amazon-kddcup24.github.io/ 获取。
原文摘要 · Abstract (English)
Online shopping is a complex multi-task, few-shot learning problem with a wide and evolving range of entities, relations, and tasks. However, existing models and benchmarks are commonly tailored to specific tasks, falling short of capturing the full complexity of online shopping. Large Language Models (LLMs), with their multi-task and few-shot learning abilities, have the potential to profoundly transform online shopping by alleviating task-specific engineering efforts and by providing users with interactive conversations. Despite the potential, LLMs face unique challenges in online shopping, such as domain-specific concepts, implicit knowledge, and heterogeneous user behaviors. Motivated by the potential and challenges, we propose Shopping MMLU, a diverse multi-task online shopping benchmark derived from real-world Amazon data. Shopping MMLU consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality, and can thus comprehensively evaluate the abilities of LLMs as general shop assistants. With Shopping MMLU, we benchmark over 20 existing LLMs and uncover valuable insights about practices and prospects of building versatile LLM-based shop assistants. Shopping MMLU can be publicly accessed at https://github.com/KL4805/ShoppingMMLU. In addition, with Shopping MMLU, we host a competition in KDD Cup 2024 with over 500 participating teams. The winning solutions and the associated workshop can be accessed at our website https://amazon-kddcup24.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。