构建可迭代优化的购物助手,解决多轮对话评估与多智能体协同难题
Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants
- 设计分维度评估体系与人工对齐的判别模型,精准衡量购物助手表现
- 提出子智能体与系统级两种优化策略,显著提升多轮交互质量
- 开源评估模板与设计指南,助力生产级对话系统落地
对话式购物助手(CSAs)是智能代理的重要应用,但从原型到生产面临两大未充分探索的挑战:如何评估多轮交互,以及如何优化紧密耦合的多智能体系统。生鲜购物进一步加剧了这些难题,因用户需求常不明确、高度依赖偏好,且受预算和库存限制。本文提出一套实用蓝图,用于评估与优化对话式购物助手,以一个生产规模的AI生鲜购物助手为例进行验证。我们构建了一个多维度评估量表,将端到端购物质量分解为结构化维度,并开发了与人工标注对齐的校准型大模型判别流水线。在此评估基础上,我们研究了两种互补的提示优化策略:(1) 子智能体GEPA,针对各智能体节点使用局部量表进行优化;(2) MAMuT(多智能体多轮)GEPA,一种新颖的系统级方法,通过多轮模拟与轨迹级评分联合优化跨智能体提示。我们公开评估量表模板与设计指导,支持从业者构建生产级CSA。
原文摘要 · Abstract (English)
Conversational shopping assistants (CSAs) represent a compelling application of agentic AI, but moving from prototype to production reveals two underexplored challenges: how to evaluate multi-turn interactions and how to optimize tightly coupled multi-agent systems. Grocery shopping further amplifies these difficulties, as user requests are often underspecified, highly preference-sensitive, and constrained by factors such as budget and inventory. In this paper, we present a practical blueprint for evaluating and optimizing conversational shopping assistants, illustrated through a production-scale AI grocery assistant. We introduce a multi-faceted evaluation rubric that decomposes end-to-end shopping quality into structured dimensions and develop a calibrated LLM-as-judge pipeline aligned with human annotations. Building on this evaluation foundation, we investigate two complementary prompt-optimization strategies based on a SOTA prompt-optimizer called GEPA (Shao et al., 2025): (1) Sub-agent GEPA, which optimizes individual agent nodes against localized rubrics, and (2) MAMuT (Multi-Agent Multi-Turn) GEPA (Herrera et al., 2026), a novel system-level approach that jointly optimizes prompts across agents using multi-turn simulation and trajectory-level scoring. We release rubric templates and evaluation design guidance to support practitioners building production CSAs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。