让客服大模型在不越界的前提下，基于证据持续优化回答质量。
Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents
- 多源检索+重排序构建可审计的问答证据库
- 通过追踪诊断失败原因，实现精准优化
- 适合需要安全可控的工业级客服系统
生产环境中的客服机器人需在迭代中持续提升回答质量，但大语言模型必须遵守证据边界、政策规则和人工介入机制。本文提出一个实际部署的「证据锚定型客服代理工作流」。通过BM25召回、问题标题向量召回、问题描述向量召回、加权RRF融合与交叉编码器重排序，构建受控的FAQ证据。策略引导的编排将此RAG证据与场景规则、对话记忆及澄清状态结合，运行于固定LangGraph DAG中。论文贡献三个可复用的部署模式：（1）混合RAG证据构建，实现多通道检索与重排序的可审计候选生成；（2）证据锚定的问题/动作决策，从结构化问答证据与场景规则中选择合适动作；（3）追踪驱动的RAG与重排序优化，通过日志追踪定位失败来源（召回、排序、候选选择、澄清、规则证据或动作策略），并评估重排序微调不仅看域内性能提升，还关注遗忘风险。
原文摘要 · Abstract (English)
Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。