大模型时代，复杂鲁棒训练对RAG的提升越来越小。
On the Diminishing Returns of Complex Robust RAG Training in the Era of Powerful LLMs
- 用简单训练法在大模型上也能达到高鲁棒性
- 模型越大，复杂训练带来的性能增益越低
- 适合追求高效部署的工程团队参考
传统RAG系统采用复杂训练策略以增强对检索噪声的鲁棒性。本文通过多模型规模和问答数据集的系统评估发现：随着模型容量增大，复杂训练策略的边际鲁棒性收益显著下降。较小模型从复杂文档选择和对抗目标中获益明显，而更强大模型在简单训练下即可实现相当或更优表现。进一步分析表明，强模型即使在简单训练下也具备更好的置信度校准、跨数据集泛化能力和更有效的注意力模式。结果表明，随着基础模型发展，复杂鲁棒训练的投入可能带来递减回报，简化RAG流程对大模型已足够维持竞争力。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) systems traditionally employ sophisticated training strategies to enhance robustness against retrieval noise. In this work, we investigate a critical question: does the benefit of these complex robust training methods diminish as language models become more powerful? Through systematic evaluation across multiple model scales and question-answering datasets, our analysis reveals a consistent trend: \emph{the marginal robustness benefit of sophisticated training strategies decreases substantially as model capacity increases.} While smaller models show significant performance improvements from complex document selection and adversarial objectives, more capable models achieve comparable or even superior performance with simpler training approaches. Further investigation demonstrates that stronger models naturally exhibit better confidence calibration, cross-dataset generalization capability, and more effective attention patterns, even under simple training regimes. These findings suggest that as foundation models evolve, the engineering effort invested in complex robust training may yield diminishing returns, indicating that simplified RAG pipelines could suffice for powerful models while maintaining competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。