让大模型在长文本中专注关键信息，避免被无关内容干扰。
Reducing Distraction in Long-Context Language Models by Focused Learning
- 用检索增强输入，提取最相关段落作为训练数据。
- 引入对比学习，使原上下文与检索片段的输出对齐。
- 在多文档问答任务上显著提升长文本理解能力。
大型语言模型（LLMs）在处理长上下文方面取得了显著进展，但有效利用长上下文仍面临注意力分散问题：无关信息占据主导，导致模型忽略关键内容。为解决此问题，我们提出一种新型训练方法，通过检索增强的数据扩充与对比学习相结合，提升模型识别相关信息的能力。在微调过程中，使用检索器提取最相关的文本片段作为增强输入，并引入辅助对比学习目标，确保原始上下文与检索子上下文的输出高度一致。在单文档和多文档问答基准测试上的大量实验表明，该方法能有效缓解注意力分散问题，显著提升模型在长文本中的表现。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have significantly enhanced their capacity to process long contexts. However, effectively utilizing this long context remains a challenge due to the issue of distraction, where irrelevant information dominates lengthy contexts, causing LLMs to lose focus on the most relevant segments. To address this, we propose a novel training method that enhances LLMs' ability to discern relevant information through a unique combination of retrieval-based data augmentation and contrastive learning. Specifically, during fine-tuning with long contexts, we employ a retriever to extract the most relevant segments, serving as augmented inputs. We then introduce an auxiliary contrastive learning objective to explicitly ensure that outputs from the original context and the retrieved sub-context are closely aligned. Extensive experiments on long single-document and multi-document QA benchmarks demonstrate the effectiveness of our proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。