提出CORD模型，解决检索生成中的位置偏倚问题。
CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation
- 通过扰动检索段落顺序增强训练一致性
- 自适应采样扰动，平衡一致性与原始排序重要性
- 在多个RAG基准上表现优于现有方法
随着检索增强生成（RAG）的应用，大语言模型需基于检索到的上下文进行生成。然而，语言模型存在位置偏倚，难以均匀关注所有上下文。先前工作通过扰动真实段落的位置合成多样化训练数据来缓解此问题。本文提出一致性和排序蒸馏的平衡方法（CORD），首先对每个训练实例进行位置扰动以增强预测一致性；同时蒸馏该对样本的行为，但在某些需要保留检索器原始排序的场景中可能适得其反。因此，CORD从插值空间中自适应采样受控噪声扰动，兼顾一致性与排序先验。实验证明，这种平衡使CORD在多种RAG基准上持续领先。
原文摘要 · Abstract (English)
With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation. CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。