arXiv:2604.09285cs.AI2026-04被引 2

构建多智能体评估框架,精准检验客服大模型的逻辑执行与应答合规性。

SAGE: A Service Agent Graph-guided Evaluation Benchmark

  • 将非结构化服务流程转为动态对话图,实现路径覆盖与逻辑合规的精确验证。
  • 在6个工业场景测试27个大模型,发现意图识别准确但后续动作执行失败的‘执行鸿沟’。
  • 支持跨领域快速部署,适合评估真实客服场景中大模型的可靠性与抗干扰能力。

大型语言模型的发展推动了客户服务自动化,但性能评估仍面临挑战。现有基准多依赖静态范式和单一维度指标,无法反映用户行为多样性或实际部署中对标准化操作流程(SOP)的严格遵循。为此,我们提出 SAGE(Service Agent Graph-guided Evaluation),一个面向自动化的双轴评估通用多智能体基准。SAGE 将非结构化 SOP 转化为动态对话图,实现逻辑合规性精准验证与完整路径覆盖。引入对抗性意图分类体系与模块化扩展机制,支持低成本跨领域部署,并实现自动化对话数据生成。评估通过裁判智能体与规则引擎分析用户与服务智能体交互,生成确定性真实标签。在6个工业场景中对27个大模型的实验揭示显著的‘执行鸿沟’:模型能准确识别意图,却无法推导正确后续动作。同时观察到‘共情韧性’现象——在高对抗强度下,模型仍维持礼貌表象,尽管底层逻辑已失效。代码与资源见 https://anonymous.4open.science/r/SAGE-Bench-4CD3/。

原文摘要 · Abstract (English)

The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominantly rely on static paradigms and single-dimensional metrics, failing to account for diverse user behaviors or the strict adherence to structured Standard Operating Procedures (SOPs) required in real-world deployments. To bridge this gap, we propose SAGE (Service Agent Graph-guided Evaluation), a universal multi-agent benchmark for automated, dual-axis assessment. SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance and comprehensive path coverage. We introduce an Adversarial Intent Taxonomy and a modular Extension Mechanism, enabling low-cost deployment across domains and facilitating automated dialogue data synthesis. Evaluation is conducted via a framework where Judge Agents and a Rule Engine analyze interactions between User and Service Agents to generate deterministic ground truth. Extensive experiments on 27 LLMs across 6 industrial scenarios reveal a significant ``Execution Gap'' where models accurately classify intents but fail to derive correct subsequent actions. We also observe ``Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures under high adversarial intensity. Code and resources are available at https://anonymous.4open.science/r/SAGE-Bench-4CD3/.

大模型评估客服自动化多智能体逻辑验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。