通过插入停顿标记，让大模型更好理解长文本中间内容。
Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration
- 用人工停顿符分割长文本，引导模型重新分配注意力。
- 在128K令牌的长文本中，3B模型提升10.61%，8B模型提升3.57%。
- 轻量级微调，适合需要长文本理解的场景。
大语言模型在理解任务上表现优异，但在处理长上下文时仍面临挑战，尤其对中间位置信息的捕捉能力不足，即‘丢失在中间’（LITM）问题。为解决此问题，我们提出暂停调优（pause-tuning），通过在数据集中人工插入停顿标记，将长输入分段，从而优化模型注意力分布。我们在针堆找针（Needle-in-a-Haystack）基准上评估该方法，要求模型从长达128K token的上下文中检索信息。实验表明，LLaMA 3.2 3B Instruct和LLaMA 3.1 8B Instruct模型平均性能分别提升10.61%和3.57%，证明该方法有效增强长上下文记忆与理解。代码与数据已公开于https://anonymous.4open.science/r/LITM-PauseTokens-7357。
原文摘要 · Abstract (English)
LLMs have demonstrated remarkable proficiency in understanding tasks but continue to struggle with long-context comprehension, particularly with content located in the middle of extensive inputs. This limitation, known as the Lost-in-the-Middle (LITM) problem, hinders models from fully processing and utilizing information across lengthy contexts. To address this issue, we introduce pause-tuning, a technique that redistributes attention to enhance comprehension of long-context inputs. Our approach involves fine-tuning language models on datasets with artificially inserted pause tokens, which serve to segment the input into smaller, more manageable parts. We evaluate pause-tuning against alternative approaches using the Needle-in-a-Haystack benchmark, where models must retrieve information embedded within contexts of up to 128K tokens. Experimental results demonstrate significant performance gains, with the LLaMA 3.2 3B Instruct model and the LLaMA 3.1 8B Instruct model improving by 10.61% and 3.57% respectively on average, suggesting that pause-tuning successfully enhances attention redistribution and improves long-context retention. The code and data are available at https://anonymous.4open.science/r/LITM-PauseTokens-7357.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。