让大模型在不增加内存的情况下记住长文本，还能保持因果关系。
Absorber LLM: Harnessing Causal Synchronization for Test-Time Training
- 通过因果同步机制，将历史信息吸收进模型参数中
- 在长文本推理任务上降低内存占用并提升准确率
- 适合需要持续处理长序列的实时应用
Transformer 模型因自注意力计算成本随序列长度增长而难以处理长序列，导致内存不足。虽然 RNN、SSM 等常数内存方法能压缩历史状态，但会丢失长尾依赖；而将上下文记忆到参数中的测试时训练（TTT）方法易过拟合于词级投影，无法保留预训练大模型的因果效应。本文提出 Absorber LLM，将长上下文保持问题建模为自监督因果同步：在将历史上下文吸收进参数后，无上下文模型应与原模型在完整上下文下的未来生成结果一致。通过同步更新后模型与原始模型的内部行为实现该目标，确保上下文吸收与泛化能力。在长上下文与流式基准测试中，Absorber LLM 显著降低推理内存并优于现有参数记忆类方法。
原文摘要 · Abstract (English)
Transformers suffer from a high computational cost that grows with sequence length for self-attention, making inference in long streams prohibited by memory consumption. Constant-memory alternatives such as RNNs and SSMs compress history into states with fixed size and thus lose long-tail dependencies, while methods that memorize contexts into parameters, such as Test-Time Training (TTT), are prone to overfitting token-level projection and fail to preserve the causal effect of context in pretrained LLMs. We propose Absorber LLM, which formulates long-context retention as a self-supervised causal synchronization: after absorbing historical contexts into parameters, a contextless model should match the original model with full context on future generations. We optimize this objective by synchronizing internal behaviors of the updated model with the original one, ensuring context absorption and generalization. Experiments on long-context and streaming benchmarks show that Absorber LLM reduces inference memory and improves accuracy over prior parameter-as-memory baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。