构建9.9万条反事实实体替换数据,训练更忠实于检索内容的RAG模型
Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models
- 通过替换原文实体制造上下文与记忆冲突,生成可控训练数据
- 99,094条样本经四重自动化质检,通过率100%
- 专为提升RAG模型上下文忠实性设计,适合模型训练与评估
检索增强生成(RAG)模型常依赖参数记忆而非检索到的上下文,削弱了检索增强的核心优势。其主要瓶颈在于缺乏显式要求模型优先使用上下文而非内部知识的训练数据。我们提出Faithfulness-QA,一个包含99,094个样本的大规模数据集,通过反事实实体替换构建而成。基于SQuAD和TriviaQA两个提取式问答基准,自动识别上下文中的答案实体,用从8类共76,953个预选实体中选出的同类型替代品进行替换,从而制造上下文与参数记忆间的可控冲突。严格的质量过滤确保在随机抽取的200个样本上,四项自动化检测均达到100%通过率。我们发布完整数据集、构建流程及实体库。该数据集旨在用于基于注意力的忠实性目标训练,并作为评估RAG系统上下文依附性的基准。数据与代码见https://github.com/qzhangFDU/faithfulness-qa-dataset。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) models frequently produce answers grounded in parametric memory rather than the retrieved context, undermining the core promise of retrieval augmentation. A fundamental obstacle to fixing this unfaithfulness is the lack of training data that explicitly requires models to prefer context over internal knowledge. We introduce Faithfulness-QA, a large-scale dataset of 99,094 samples constructed through counterfactual entity substitution. Starting from two established extractive QA benchmarks--SQuAD and TriviaQA--we automatically identify answer-bearing named entities in each context, replace them with type-consistent alternatives drawn from a curated bank of 76,953 entities, and thereby manufacture controlled knowledge conflicts between context and parametric memory. Rigorous quality filtering ensures 100% pass rates across four automated checks on random 200-sample audits. We release the full dataset, the construction pipeline, and a typed entity bank covering eight named entity categories. Faithfulness-QA is designed as a training resource for attention-based faithfulness objectives and as an evaluation benchmark for measuring context-grounding behavior in RAG systems. Data and code are available at https://github.com/qzhangFDU/faithfulness-qa-dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。