让人工反馈直接嵌入客服流程，持续优化大模型效果
Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support
- 将四种实时反馈信号融入实际客服操作中
- 召回率提升11.7%，生成质量提高8.4%，采纳率升4.5%
- 适合需要持续迭代的智能客服系统团队
我们提出一种Agent-in-the-Loop（AITL）框架，构建持续的数据飞轮，用于迭代优化基于大语言模型的客户支持系统。与依赖批量标注的离线方法不同，AITL将四种关键标注直接集成到真实客户运营中：(1) 响应偏好对比，(2) 代理采纳及理由，(3) 知识相关性判断，(4) 缺失知识识别。这些反馈信号无缝回流至模型更新，将重训练周期从数月缩短至数周。在美国客服团队的生产试点中，系统在检索准确率上实现+11.7% recall@75、+14.8% precision@8，生成质量提升+8.4%有助于度，代理采纳率增加+4.5%。结果表明，将人工反馈直接嵌入工作流，能有效持续优化大模型客服系统。
原文摘要 · Abstract (English)
We introduce an Agent-in-the-Loop (AITL) framework that implements a continuous data flywheel for iteratively improving an LLM-based customer support system. Unlike standard offline approaches that rely on batch annotations, AITL integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. These feedback signals seamlessly feed back into models' updates, reducing retraining cycles from months to weeks. Our production pilot involving US-based customer support agents demonstrated significant improvements in retrieval accuracy (+11.7% recall@75, +14.8% precision@8), generation quality (+8.4% helpfulness) and agent adoption rates (+4.5%). These results underscore the effectiveness of embedding human feedback loops directly into operational workflows to continuously refine LLM-based customer support system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。