arXiv:2601.00596cs.CL2026-01Conference of the …被引 10

评测大模型客服代理在复杂业务规则下的合规表现,提出新基准与评估指标。

Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

  • 用图结构生成真实客服场景,模拟多步政策流程。
  • 动态提示代理使小模型在政策遵守上超越大模型。
  • 适合研究智能客服合规性与系统设计的开发者和学者。

传统客服系统如交互式语音应答(IVR)依赖固定脚本,难以应对复杂、政策驱动的任务。尽管大语言模型(LLM)代理提供了潜在替代方案,但评估其在真实支持流程中遵守业务规则的能力仍是挑战。现有基准主要关注工具使用或任务完成率,忽略了代理在多步骤政策、任务依赖关系中的表现,以及对用户或环境异常行为的鲁棒性。本文提出JourneyBench,一个用于评估政策感知代理的基准。该基准利用图表示生成多样且真实的客服场景,并引入用户旅程覆盖度(User Journey Coverage Score)作为衡量政策遵守的新指标。我们在三个领域共703次对话中评估了多个先进LLM,采用静态提示代理(SPA)和动态提示代理(DPA)两种设计。结果表明,DPA显著提升政策遵守率,甚至使GPT-4o-mini在某些任务中优于GPT-4o。研究证实结构化调度的重要性,并确立JourneyBench作为推动客服智能化超越IVR时代的关键资源。

原文摘要 · Abstract (English)

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.

客服代理政策遵守基准测试LLM应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。