arXiv:2602.15859cs.CL2026-02

从通话记录构建可信赖的智能客服,自动处理三成电话且准确率接近完美。

From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants

  • 用评分筛选高质量对话,再用大模型提取结构化知识用于AI问答
  • 在房地产与招聘领域实现30%电话自动处理,事实准确率近乎完美
  • 支持对抗测试,适合高要求场景如实时信息依赖型服务

面向客户交互行业,构建可靠对话式AI助手仍面临噪声数据、知识碎片化及精准人工转接需求等挑战,尤其在依赖实时信息的领域。本文提出一个端到端框架,直接从历史通话记录构建并评估对话式AI助手。首先使用简化的PIPA框架对通话记录进行评分,聚焦观察一致性和恰当响应行为,筛选出高质量、连贯性强且人工应答有效的互动。随后利用大语言模型(LLMs)从精选对话中提取结构化知识,并作为检索增强生成(RAG)管道的唯一知识源。通过系统性提示调优,逐步实现从单体提示到轻量、模块化、受控设计,保障行为一致性、安全性和可控性。评估采用基于通话记录的用户模拟器,量化衡量通话覆盖度、事实准确性及人工介入行为;额外红队测试评估其对提示注入、超范围和上下文无关攻击的鲁棒性。实验在房地产和专业招聘领域进行,这两个领域因依赖实时数据而自动化程度低且极具挑战性。尽管如此,该助手仍能自主处理约30%的通话,实现近似完美的事实准确率和拒绝行为,并在对抗测试中表现出强鲁棒性。

原文摘要 · Abstract (English)

Building reliable conversational AI assistants for customer-facing industries remains challenging due to noisy conversational data, fragmented knowledge, and the requirement for accurate human hand-off - particularly in domains that depend heavily on real-time information. This paper presents an end-to-end framework for constructing and evaluating a conversational AI assistant directly from historical call transcripts. Incoming transcripts are first graded using a simplified adaptation of the PIPA framework, focusing on observation alignment and appropriate response behavior, and are filtered to retain only high-quality interactions exhibiting coherent flow and effective human agent responses. Structured knowledge is then extracted from curated transcripts using large language models (LLMs) and deployed as the sole grounding source in a Retrieval-Augmented Generation (RAG) pipeline. Assistant behavior is governed through systematic prompt tuning, progressing from monolithic prompts to lean, modular, and governed designs that ensure consistency, safety, and controllable execution. Evaluation is conducted using a transcript-grounded user simulator, enabling quantitative measurement of call coverage, factual accuracy, and human escalation behavior. Additional red teaming assesses robustness against prompt injection, out-of-scope, and out-of-context attacks. Experiments are conducted in the Real Estate and Specialist Recruitment domains, which are intentionally challenging and currently suboptimal for automation due to their reliance on real-time data. Despite these constraints, the assistant autonomously handles approximately 30 percents of calls, achieves near-perfect factual accuracy and rejection behavior, and demonstrates strong robustness under adversarial testing.

对话系统RAG知识提取智能客服

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。