通过只处理输入关键部分，显著提升长文本循环模型性能。
Overflow Prevention Enhances Long-Context Recurrent LLMs
- 用分块推理只处理输入中最相关部分，缓解记忆溢出问题。
- 在LongBench上使Falcon3-Mamba-Inst-7B等模型性能提升14%至51%。
- 方法简单有效，适合追求长上下文效率的开发者使用。
近期大语言模型趋势是发展循环的亚二次模型以提升长上下文处理效率。我们研究了主流的大规模长上下文模型，聚焦其固定大小的循环记忆对性能的影响。实验发现，即便这些模型经过长时间上下文训练，仍未能充分利用长上下文。具体而言，我们提出一种基于分块的推理方法，仅识别并处理输入中最相关的部分，可有效缓解循环记忆失败问题，并在多个长上下文任务中表现优异：在LongBench上，该方法使Falcon3-Mamba-Inst-7B、Falcon-Mamba-Inst-7B、RecurrentGemma-IT-9B和RWKV6-Finch-7B的性能分别提升14%、28%、50%和51%。令人意外的是，该简单策略在具有挑战性的LongBench v2基准上也达到顶尖水平，表现可与同等规模的Transformer模型媲美。此外，我们的发现质疑了循环模型是否真正利用了长程依赖关系，因为单块策略在本应需要跨上下文关联的任务中仍表现更优。
原文摘要 · Abstract (English)
A recent trend in LLMs is developing recurrent sub-quadratic models that improve long-context processing efficiency. We investigate leading large long-context models, focusing on how their fixed-size recurrent memory affects their performance. Our experiments reveal that, even when these models are trained for extended contexts, their use of long contexts remains underutilized. Specifically, we demonstrate that a chunk-based inference procedure, which identifies and processes only the most relevant portion of the input can mitigate recurrent memory failures and be effective for many long-context tasks: On LongBench, our method improves the overall performance of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%, RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Surprisingly, this simple approach also leads to state-of-the-art results in the challenging LongBench v2 benchmark, showing competitive performance with equivalent size Transformers. Furthermore, our findings raise questions about whether recurrent models genuinely exploit long-range dependencies, as our single-chunk strategy delivers stronger performance - even in tasks that presumably require cross-context relations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。