用段落不确定性提升长文本生成效果，仅用4%数据达顶尖性能
UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation
- 基于信噪比估算文本块不确定性，优化检索相似度
- 在LLaMA-2-7B上提升2.03%，仅需4%训练数据
- 无需微调，适配不同上下文长度的大模型
我们提出UncertaintyRAG，一种新型长上下文检索增强生成方法，利用基于信噪比（SNR）的段落级不确定性来估计文本块间的相似性。该不确定性提升模型校准能力，增强鲁棒性，并缓解随机分块带来的语义不一致问题。基于此，我们设计了一种高效的无监督学习训练方法，结合有效的数据采样与扩展策略。UncertaintyRAG在LLaMA-2-7B上相比基线提升2.03%，达到当前最优结果，且仅使用其他先进开源检索模型所需训练数据的4%，在分布外设置下表现优异。该方法通过段落不确定性实现良好校准，显著提升长上下文RAG任务的泛化与鲁棒性。此外，其轻量级检索模型可无缝集成至任意大语言模型，支持不同上下文窗口长度，无需微调，展现方法的高度灵活性。
原文摘要 · Abstract (English)
We present UncertaintyRAG, a novel approach for long-context Retrieval-Augmented Generation (RAG) that utilizes Signal-to-Noise Ratio (SNR)-based span uncertainty to estimate similarity between text chunks. This span uncertainty enhances model calibration, improving robustness and mitigating semantic inconsistencies introduced by random chunking. Leveraging this insight, we propose an efficient unsupervised learning technique to train the retrieval model, alongside an effective data sampling and scaling strategy. UncertaintyRAG outperforms baselines by 2.03% on LLaMA-2-7B, achieving state-of-the-art results while using only 4% of the training data compared to other advanced open-source retrieval models under distribution shift settings. Our method demonstrates strong calibration through span uncertainty, leading to improved generalization and robustness in long-context RAG tasks. Additionally, UncertaintyRAG provides a lightweight retrieval model that can be integrated into any large language model with varying context window lengths, without the need for fine-tuning, showcasing the flexibility of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。