让大模型突破上下文长度限制，像人一样记住长期信息
InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation
- 将临时上下文转化为永久参数更新，模拟人类长短时记忆
- 上下文长度减少90%，性能达全量输入的103%
- 适合处理超长文本任务，提升模型可扩展性
在上下文学习（ICL）中，大语言模型的表现受限于有限的上下文窗口，尤其在超长输入场景下。为此，我们提出InfiniteICL框架，借鉴人类认知系统中的短时与长时记忆机制，将临时上下文知识转化为持久的参数更新。该方法显著降低内存消耗，在不同输入长度下保持鲁棒性能，并通过上下文知识提取、选择与巩固原理，理论上实现无限上下文整合。实验表明，该方法在事实回忆、基于事实推理和技能习得任务中，将上下文长度压缩90%的同时，平均性能达到全上下文提示的103%。在长达200万词元的真实复杂上下文上进行多轮序列变换时，仅使用原始上下文0.4%的量，性能仍优于全上下文提示。结果表明，InfiniteICL有望突破传统上下文窗口限制，提升大模型的可扩展性与效率。
原文摘要 · Abstract (English)
In-context learning (ICL) is critical for large language models (LLMs), but its effectiveness is constrained by finite context windows, particularly in ultra-long contexts. To overcome this, we introduce InfiniteICL, a framework that parallels context and parameters in LLMs with short- and long-term memory in human cognitive systems, focusing on transforming temporary context knowledge into permanent parameter updates. This approach significantly reduces memory usage, maintains robust performance across varying input lengths, and theoretically enables infinite context integration through the principles of context knowledge elicitation, selection, and consolidation. Evaluations demonstrate that our method reduces context length by 90% while achieving 103% average performance of full-context prompting across fact recall, grounded reasoning, and skill acquisition tasks. When conducting sequential multi-turn transformations on complex, real-world contexts (with length up to 2M tokens), our approach surpasses full-context prompting while using only 0.4% of the original contexts. These findings highlight InfiniteICL's potential to enhance the scalability and efficiency of LLMs by breaking the limitations of conventional context window sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。