LoLA让Transformer在长文本中高效记忆，准确率提升至97.4%
LoLA: Low-Rank Linear Attention With Sparse Caching
- 用三种内存系统分层存储历史键值对，提升记忆效率
- 在4K上下文下，缓存大小仅为Llama-3.1的1/4.6，准确率达97.4%
- 无需训练，适合长期上下文学习与常识推理场景
Transformer推理的每标记开销随上下文长度增长，限制了其在终身上下文学习中的应用。线性注意力是一种高效替代方案，可在无限上下文长度下保持恒定内存占用。尽管具备潜力，但其记忆容量不足。本文提出LoLA，一种无需训练的线性注意力增强方法，显著提升关联回忆能力。LoLA将上下文中的历史键值对分配至三个记忆系统：(i) 局部滑动窗口缓存中的近期对；(ii) 全局稀疏缓存中的难记对；(iii) 线性注意力递归隐藏状态中的通用对。通过消融实验表明，自回忆误差指标对有效管理长期关联记忆至关重要。在传递键检索任务中，LoLA将基础模型准确率从0.6%提升至97.4%。该效果在4K上下文长度下实现，缓存规模仅为Llama-3.1 8B的4.6倍小。LoLA还在零样本常识推理任务中优于其他1B和8B参数的次二次模型。
原文摘要 · Abstract (English)
The per-token cost of transformer inference scales with context length, preventing its application to lifelong in-context learning. Linear attention is an efficient alternative that maintains a constant memory footprint, even on infinite context lengths. While this is a potential candidate for lifelong learning, it falls short in memory capacity. In this paper, we propose LoLA, a training-free augmentation to linear attention that boosts associative recall. LoLA distributes past key-value pairs from context into three memory systems: (i) recent pairs in a local sliding window cache; (ii) difficult-to-memorize pairs in a sparse, global cache; and (iii) generic pairs in the recurrent hidden state of linear attention. We show through ablations that our self-recall error metric is crucial to efficiently manage long-term associative memories. On pass-key retrieval tasks, LoLA improves the base model's performance from 0.6% to 97.4% accuracy. This is achieved with a 4.6x smaller cache than Llama-3.1 8B on 4K context length. LoLA also outperforms other 1B and 8B parameter subquadratic models on zero-shot commonsense reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。