对比微调与检索增强在多跳问答中的表现,发现检索更适配新知识。
Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge
- 比较微调与检索增强两种知识注入方式
- 检索增强在新知识上提升显著,监督微调整体最准
- 适合需要外部或组合知识的场景
多跳问答广泛用于评估大语言模型的推理能力,需整合多条支持性知识才能得出正确答案。尽管已有研究探索了向LLM提供知识的不同机制,如微调和检索增强生成(RAG),但它们在多跳问答中的相对有效性仍不明确,尤其是在所需知识为时间上新颖的情况下。本文系统比较了参数化与非参数化知识注入方法在开放域多跳问答中的表现。在三个70亿参数的开源LLM上,评估了无监督微调(持续预训练)、有监督微调和检索增强生成。实验基于两个基准:QASC——标准的多跳科学问答数据集,以及一个新构建的、包含超过10,000个来自2024年维基百科事件的多跳问题数据集,用于测试模型超出其预训练截止时间的知识。结果表明,无监督微调对基础模型的提升有限,说明仅靠持续预训练不足以提高多跳推理准确率。相比之下,检索增强生成在回答依赖时间新知识的问题时带来显著且一致的改进。有监督微调在所有模型和数据集上达到最高总体准确率。这些发现揭示了不同知识注入机制在支持多跳问答上的根本差异,强调了在需要外部或组合知识时,检索方法的重要性。
原文摘要 · Abstract (English)
Multi-hop question answering is widely used to evaluate the reasoning capabilities of large language models (LLMs), as it requires integrating multiple pieces of supporting knowledge to arrive at a correct answer. While prior work has explored different mechanisms for providing knowledge to LLMs, such as finetuning and retrieval-augmented generation (RAG), their relative effectiveness for multi-hop question answering remains insufficiently understood, particularly when the required knowledge is temporally novel. In this paper, we systematically compare parametric and non-parametric knowledge injection methods for open-domain multi-hop question answering. We evaluate unsupervised fine-tuning (continual pretraining), supervised fine-tuning, and retrieval-augmented generation across three 7B-parameter open-source LLMs. Experiments are conducted on two benchmarks: QASC, a standard multi-hop science question answering dataset, and a newly constructed dataset of over 10,000 multi-hop questions derived from Wikipedia events in 2024, designed to test knowledge beyond the models' pretraining cutoff. Our results show that unsupervised fine-tuning provides only limited gains over base models, suggesting that continual pretraining alone is insufficient for improving multi-hop reasoning accuracy. In contrast, retrieval-augmented generation yields substantial and consistent improvements, particularly when answering questions that rely on temporally novel information. Supervised fine-tuning achieves the highest overall accuracy across models and datasets. These findings highlight fundamental differences in how knowledge injection mechanisms support multi-hop question answering and underscore the importance of retrieval-based methods when external or compositional knowledge is required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。