用问答数据构建专用奖励模型，让大模型生成更符合上下文的回应。
RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generation
- 将问答数据转为偏好对，聚焦生成内容与检索信息的一致性。
- 4000样本训练出的模型在评测中超越70B以上大模型15.5%。
- 适合需要精准、可信回答的场景，如医疗或法律咨询。
现有奖励模型多基于通用偏好数据训练，在检索增强生成（RAG）场景下表现不佳，难以判断回复是否忠实于检索到的上下文、是否相关、是否恰当拒绝无关提问，以及信息的完整性和简洁性。为解决缺乏公开的RAG专用偏好数据集和专门模型的问题，我们提出RAGferee方法，将问答数据转化为强调事实一致性的偏好对，用于训练更适配RAG任务的上下文感知奖励模型。基于此方法，我们构建了包含4000个样本的小型偏好数据集，并微调了参数量从70亿到240亿的奖励模型。实验表明，这些基于RAGferee训练的模型在ContextualJudgeBench评测中达到领先性能，显著优于使用多达240万样本的通用语料训练的700亿以上模型，绝对提升达+15.5%。
原文摘要 · Abstract (English)
Existing Reward Models (RMs), typically trained on general preference data, struggle in Retrieval Augmented Generation (RAG) settings, which require judging responses for faithfulness to retrieved context, relevance to the user query, appropriate refusals when context is insufficient, completeness and conciseness of information. To address the lack of publicly available RAG-centric preference datasets and specialised RMs, we introduce RAGferee, a methodology that repurposes question-answering (QA) datasets into preference pairs that prioritise groundedness over stylistic features, enabling the training of contextual RMs better suited to judging RAG responses. Using RAGferee, we curate a small preference dataset of 4K samples and fine-tune RMs ranging from 7B to 24B parameters. Our RAG-centric RMs achieve state-of-the-art performance on ContextualJudgeBench, surpassing existing 70B+ RMs trained on much larger (up to 2.4M samples) general corpora, with an absolute improvement of +15.5%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。