arXiv:2510.27051cs.AIcs.LG2025-10Conference of the …被引 2

用闭环控制提升企业AI助手,自动修复检索错误并持续优化性能。

Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement

  • 通过MAPE控制环构建数据飞轮,实现故障自检与修复。
  • 3个月收集495条负样本,定位路由与重写错误并分别优化。
  • 适合需要持续迭代的大型企业AI系统部署参考。

企业AI代理需持续适应以保持准确性、降低延迟并贴合用户需求。我们实现了NVInfo AI中基于MAPE控制环的数据飞轮,该系统服务于超过30,000名员工,采用NVIDIA的Mixture-of-Experts(MoE)知识助手。通过运营这一闭环系统,我们系统性地解决检索增强生成(RAG)管道中的失败问题,并实现持续学习。在部署后3个月内,我们监测反馈并收集了495个负面样本。分析显示两大主要故障模式:路由错误(5.25%)和查询重述错误(3.2%)。利用NVIDIA NeMo微服务,我们通过微调实施针对性改进:路由方面,将Llama 3.1 70B模型替换为微调后的8B变体,准确率提升至96%,模型规模减少10倍,延迟降低70%;查询重述方面,微调带来3.7%的准确率提升和40%的延迟下降。本方法表明,当人类反馈被结构化整合进数据飞轮时,可使企业级AI代理转变为自我优化系统。关键经验包括在反馈有限情况下保障代理鲁棒性、应对隐私限制以及在生产环境中执行分阶段上线。该工作提供了一套可复用的蓝图,用于构建可在真实世界大规模使用中持续学习的稳健自适应企业级AI代理。

原文摘要 · Abstract (English)

Enterprise AI agents must continuously adapt to maintain accuracy, reduce latency, and remain aligned with user needs. We present a practical implementation of a data flywheel in NVInfo AI, NVIDIA's Mixture-of-Experts (MoE) Knowledge Assistant serving over 30,000 employees. By operationalizing a MAPE-driven data flywheel, we built a closed-loop system that systematically addresses failures in retrieval-augmented generation (RAG) pipelines and enables continuous learning. Over a 3-month post-deployment period, we monitored feedback and collected 495 negative samples. Analysis revealed two major failure modes: routing errors (5.25\%) and query rephrasal errors (3.2\%). Using NVIDIA NeMo microservices, we implemented targeted improvements through fine-tuning. For routing, we replaced a Llama 3.1 70B model with a fine-tuned 8B variant, achieving 96\% accuracy, a 10x reduction in model size, and 70\% latency improvement. For query rephrasal, fine-tuning yielded a 3.7\% gain in accuracy and a 40\% latency reduction. Our approach demonstrates how human-in-the-loop (HITL) feedback, when structured within a data flywheel, transforms enterprise AI agents into self-improving systems. Key learnings include approaches to ensure agent robustness despite limited user feedback, navigating privacy constraints, and executing staged rollouts in production. This work offers a repeatable blueprint for building robust, adaptive enterprise AI agents capable of learning from real-world usage at scale.

AI代理数据飞轮闭环优化企业应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。