用记忆机制让大模型处理更长文本，内存不变、效率更高。
Extending LLM Context via Associative Recurrent Memory
- 引入关联记忆模块，实现恒定内存占用的长文本处理。
- 模型可处理超长上下文，且在原长度内性能不降，节省30%算力。
- 适合需要长文本理解的场景，如法律、医疗文档分析。
扩展大语言模型（LLM）的上下文长度对诸多实际应用至关重要，但标准Transformer受限于二次计算复杂度和线性内存增长。本文研究了关联循环记忆变压器（ARMT）作为一种实用方法，实现LLM的长上下文处理、恒定内存扩展与更高效率。主要贡献包括：首先，构建两个领域专用的长上下文数据集，用于评估窄域微调场景下的真实工作负载；其次，提出一套完整的ARMT训练方案，结合持续预训练、合成长上下文数据生成、课程学习及选择性地将关联记忆集成到特定模型层；第三，通过大量实验表明，增强后的模型：(i) 能在超出原始上下文限制的情况下处理输入，且性能不劣于原长基线；(ii) 对分布外的上下文长度具有更强泛化能力；(iii) 在保持原上下文窗口性能的前提下，减少30%的浮点运算量（FLOPs）。
原文摘要 · Abstract (English)
Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and linear memory scaling. In this work, we investigate the Associative Recurrent Memory Transformer (ARMT) as a practical approach for enabling long-context processing in LLMs, constant memory scaling, and better efficiency. We make three main contributions. First, we construct two domain-specific long-context datasets designed to evaluate realistic workloads, focusing on narrow-domain fine-tuning scenarios. Second, we propose a comprehensive training recipe for ARMT-based context extension, combining continued pre-training, synthetic long-context data generation, curriculum learning, and selective integration of associative memory into chosen model layers. Third, we present an extensive experimental study demonstrating that ARMT-augmented models: (i) process inputs well beyond their original context limits without degrading performance relative to in-limit baselines; (ii) generalize more effectively to out-of-distribution context lengths; and (iii) need 30% less FLOPs while preserving baseline performance within the original context window.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。