arXiv:2511.04880cs.AI2025-11

让检索生成系统实时学习用户反馈,动态优化搜索结果。

DMA: Online RAG Alignment with Human Feedback

  • 分文档、列表、回复三级反馈,构建闭环学习机制
  • 线上部署提升用户参与度,线下测试在问答任务上显著领先
  • 适合需要持续优化的对话式检索系统开发者

检索增强生成(RAG)系统通常依赖静态检索,难以适应意图变化和内容漂移。本文提出动态记忆对齐(DMA),一种在线学习框架,通过多粒度人类反馈系统性地对齐交互场景下的排序结果。DMA将文档级、列表级和回复级信号整合为统一学习流程:使用监督学习训练点对点与列表级排序器,基于回复级偏好进行策略优化,并通过知识蒸馏生成轻量评分器以支持低延迟服务。文中“记忆”指模型的工作记忆,即大语言模型用于上下文学习的全部可见上下文。采用双轨评估协议模拟真实部署:(i) 大规模在线A/B测试,分离各反馈源的贡献;(ii) 在知识密集型基准上的少量样本离线测试。线上多月工业部署显示用户参与度明显提升;线下测试中,DMA在保持基础检索能力的同时,在对话式问答任务(TriviaQA、HotpotQA)上取得显著进步。整体结果表明,DMA是一种无需牺牲基线性能的、以反馈驱动的实时自适应RAG方法。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) systems often rely on static retrieval, limiting adaptation to evolving intent and content drift. We introduce Dynamic Memory Alignment (DMA), an online learning framework that systematically incorporates multi-granularity human feedback to align ranking in interactive settings. DMA organizes document-, list-, and response-level signals into a coherent learning pipeline: supervised training for pointwise and listwise rankers, policy optimization driven by response-level preferences, and knowledge distillation into a lightweight scorer for low-latency serving. Throughout this paper, memory refers to the model's working memory, which is the entire context visible to the LLM for In-Context Learning. We adopt a dual-track evaluation protocol mirroring deployment: (i) large-scale online A/B ablations to isolate the utility of each feedback source, and (ii) few-shot offline tests on knowledge-intensive benchmarks. Online, a multi-month industrial deployment further shows substantial improvements in human engagement. Offline, DMA preserves competitive foundational retrieval while yielding notable gains on conversational QA (TriviaQA, HotpotQA). Taken together, these results position DMA as a principled approach to feedback-driven, real-time adaptation in RAG without sacrificing baseline capability.

RAG在线学习人机反馈对话系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。