让大模型主动回顾并总结长文本，提升理解能力
Towards robust long-context understanding of large language model via active recap learning
- 通过损失差值识别关键段落，用LLM生成摘要
- 推理时自动调用回顾摘要，实现跨段落记忆
- 在RULER和LongBench上分别提升26.8%和9.44%
本文提出主动回顾学习（ARL），一种增强大语言模型长文本理解能力的框架。ARL在持续预训练中通过损失差距识别长上下文中的关键标记，并基于前文段落生成摘要；在推理阶段，模型可自主生成并使用这些回溯摘要，建立跨段落的递归记忆机制。实验表明，ARL在RULER上取得26.8%的性能提升，在LongBench上提升9.44%。该方法为提升大模型长上下文理解提供了简单而有效的持续预训练方案，推动了可扩展的记忆增强技术发展。
原文摘要 · Abstract (English)
In this paper, we propose active recap learning (ARL), a framework for enhancing large language model (LLM) in understanding long contexts. ARL enables models to revisit and summarize earlier content through targeted sequence construction during contined pretraining and retrospective summarization at inference. First, we identify key tokens in prepared long context based on loss gaps between long and short forward contexts and find most revant preceding paragraphs, then summarize them using an LLM. Second, ARL equips models with the ability to autonomously generate and utilize these retrospective summaries during inference, thereby establishing a recursive memory mechanism across paragraphs. Experimental results show substantial gains, with ARL achieving a 26.8% improvement on RULER and a 9.44% improvement on LongBench. Overall, ARL offers a simple yet effective continued pretraining-based approach to strengthen long-context understanding, advancing scalable memory augmentation in LLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。