CoMeT让大模型用恒定内存处理超长文本，还能精准找关键信息。
CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling
- 用双记忆机制分层管理上下文，临时记新内容，全局记长期依赖
- 在100万词序列中仍能准确找回任意位置的密钥，32k上下文微调后有效
- 可直接接入现有模型，适合需要超长推理的智能体和问答系统
标准Transformer的二次复杂度和无限增长的键值缓存严重限制了长上下文处理。为此，我们提出协作记忆Transformer（CoMeT），一种新型架构，使大语言模型在恒定内存占用和线性时间复杂度下处理任意长度序列。作为高效即插即用模块，CoMeT仅需少量微调即可集成到预训练模型中。它对序列块进行处理，采用双记忆系统：基于先进先出队列的临时记忆用于存储近期事件，带门控更新规则的全局记忆用于捕捉长程依赖。这两个记忆共同作为下一区块的动态软提示。为实现超长上下文的高效微调，我们引入一种新的层级流水线并行策略。实验表明，经32k上下文微调的CoMeT模型，可在100万词序列中准确检索任意位置的密钥。在SCROLLS基准测试中，CoMeT优于其他高效方法，在摘要任务上表现接近全注意力基线。其实际效果在真实世界智能体和用户行为问答任务中得到验证。代码已开源：https://github.com/LivingFutureLab/Comet
原文摘要 · Abstract (English)
The quadratic complexity and indefinitely growing key-value (KV) cache of standard Transformers pose a major barrier to long-context processing. To overcome this, we introduce the Collaborative Memory Transformer (CoMeT), a novel architecture that enables LLMs to handle arbitrarily long sequences with constant memory usage and linear time complexity. Designed as an efficient, plug-in module, CoMeT can be integrated into pre-trained models with only minimal fine-tuning. It operates on sequential data chunks, using a dual-memory system to manage context: a temporary memory on a FIFO queue for recent events, and a global memory with a gated update rule for long-range dependencies. These memories then act as a dynamic soft prompt for the next chunk. To enable efficient fine-tuning on extremely long contexts, we introduce a novel layer-level pipeline parallelism strategy. The effectiveness of our approach is remarkable: a model equipped with CoMeT and fine-tuned on 32k contexts can accurately retrieve a passkey from any position within a 1M token sequence. On the SCROLLS benchmark, CoMeT surpasses other efficient methods and achieves performance comparable to a full-attention baseline on summarization tasks. Its practical effectiveness is further validated on real-world agent and user behavior QA tasks. The code is available at: https://github.com/LivingFutureLab/Comet
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。