无需标注数据，用自监督方法提升大模型问答忠实度。
SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generation
- 通过对比有无上下文时的输出，自动生成偏好数据对
- 在多个数据集上达到当前最优的忠实度表现
- 适合希望提升RAG系统可靠性且无标注成本的研究者
检索增强生成（RAG）系统要求大语言模型根据检索到的上下文生成忠实回答。然而，忠实性幻觉仍是关键挑战，现有方法常需昂贵的标注或后训练、带来显著推理开销。为此，我们提出首个自监督对齐方法——自监督忠实度优化（SSFO）。SSFO通过对比模型在有/无上下文时的输出，构建偏好数据对，利用直接偏好优化（DPO）实现对齐，无需标签成本或额外推理负担。理论与实证表明，SSFO利用了一种良性的似然偏移机制，将概率质量从参数化词元转移至与上下文一致的词元。基于此，我们设计了改进的DPO损失函数以促进该偏移。全面评估显示，SSFO在多个基于上下文的问答数据集上显著优于现有方法，达到当前最优忠实度。尤其在跨语言场景中表现出强泛化能力，同时保持通用指令遵循性能。代码与模型已公开于匿名链接：https://github.com/chkwy/SSFO
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems require Large Language Models (LLMs) to generate responses that are faithful to the retrieved context. However, faithfulness hallucination remains a critical challenge, as existing methods often require costly supervision and post-training or significant inference burdens. To overcome these limitations, we introduce Self-Supervised Faithfulness Optimization (SSFO), the first self-supervised alignment approach for enhancing RAG faithfulness. SSFO constructs preference data pairs by contrasting the model's outputs generated with and without the context. Leveraging Direct Preference Optimization (DPO), SSFO aligns model faithfulness without incurring labeling costs or additional inference burden. We theoretically and empirically demonstrate that SSFO leverages a benign form of \emph{likelihood displacement}, transferring probability mass from parametric-based tokens to context-aligned tokens. Based on this insight, we propose a modified DPO loss function to encourage likelihood displacement. Comprehensive evaluations show that SSFO significantly outperforms existing methods, achieving state-of-the-art faithfulness on multiple context-based question-answering datasets. Notably, SSFO exhibits strong generalization, improving cross-lingual faithfulness and preserving general instruction-following capabilities. We release our code and model at the anonymous link: https://github.com/chkwy/SSFO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。