小模型通过大模型推理能力蒸馏,显著提升长文本理解与信息提取能力。
Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
- 用大模型的思维链蒸馏小模型,增强其长上下文推理能力。
- 在多文档问答中,蒸馏后模型对长上下文的信息提取准确率显著提升。
- 解决长文本中段信息丢失问题,适合需要深度理解的RAG系统应用。
推理蒸馏已成为提升小型语言模型推理能力的有效方法。然而,大规模推理蒸馏对其他关键能力(尤其是上下文检索与推理)的影响尚不明确,这一空白尤为突出,因检索增强生成(RAG)系统越来越依赖高效获取和利用上下文信息以生成可靠回答。为探究长思维链过程如何影响长上下文理解,我们基于以卓越推理能力著称的 Deepseek-R1 模型,对一系列开源小模型进行了蒸馏,并在多文档问答任务中评估其从扩展上下文中提取和整合相关信息的能力。实验表明,经过蒸馏的模型在长上下文理解上表现更优。分析显示,蒸馏促使模型在上下文分析与信息解析中采用更详细、更显式的推理过程,从而增强了长上下文感知能力,有效缓解了长期困扰长上下文模型的‘中间信息丢失’问题。
原文摘要 · Abstract (English)
Reasoning distillation has emerged as an effective approach to enhance the reasoning capabilities of smaller language models. However, the impact of large-scale reasoning distillation on other critical abilities, particularly in-context retrieval and reasoning, remains unexplored. This gap in understanding is particularly significant given the increasing importance of Retrieval-Augmented Generation (RAG) systems, where efficient acquisition and utilization of contextual information are paramount for generating reliable responses. Motivated by the need to understand how the extended long-CoT process influences long-context comprehension, we conduct a comprehensive investigation using a series of open-source models distilled from Deepseek-R1, renowned for its exceptional reasoning capabilities. Our study focuses on evaluating these models' performance in extracting and integrating relevant information from extended contexts through multi-document question and answering tasks. Through rigorous experimentation, we demonstrate that distilled reasoning patterns significantly improve long-context understanding. Our analysis reveals that distillation fosters greater long-context awareness by promoting more detailed and explicit reasoning processes during context analysis and information parsing. This advancement effectively mitigates the persistent "lost in the middle" issue that has hindered long-context models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。