提出LCA方法，在压缩上下文的同时降低计算和存储开销。
Latent-Condensed Transformer for Efficient Long Context Modeling
- 在低维隐空间中直接压缩上下文，分离语义与位置信息
- 实现128K上下文下90%缓存减少和2.5倍预填充加速
- 无需额外参数，可适配多种注意力结构，适合长文本处理
大语言模型在处理长上下文时面临键值缓存线性增长和自注意力二次复杂度的挑战。现有方法分别解决这些问题：多头隐空间注意力（MLA）通过将令牌投影到低维隐空间来减少KV缓存，而稀疏注意力则降低计算量。然而，稀疏方法无法原生作用于MLA的压缩隐结构，错失联合优化机会。本文提出隐空间浓缩注意力（LCA），直接在MLA的隐空间中压缩上下文，将表示解耦为语义隐向量和位置键。LCA通过查询感知池化聚合语义向量，通过锚点选择保留位置键。该方法联合减少计算成本与KV缓存，且不增加参数。除MLA外，LCA设计架构无关，可扩展至GQA等其他注意力机制。理论上证明误差界与序列长度无关。实验显示，LCA在128K上下文下实现最高2.5倍预填充加速和90%缓存减少，同时保持竞争力性能。
原文摘要 · Abstract (English)
Large language models (LLMs) face significant challenges in processing long contexts due to the linear growth of the key-value (KV) cache and quadratic complexity of self-attention. Existing approaches address these bottlenecks separately: Multi-head Latent Attention (MLA) reduces the KV cache by projecting tokens into a low-dimensional latent space, while sparse attention reduces computation. However, sparse methods cannot operate natively on MLA's compressed latent structure, missing opportunities for joint optimization. In this paper, we propose Latent-Condensed Attention (LCA), which directly condenses context within MLA's latent space, where the representation is disentangled into semantic latent vectors and positional keys. LCA separately aggregates semantic vectors via query-aware pooling and preserves positional keys via anchor selection. This approach jointly reduces both computational cost and KV cache without adding parameters. Beyond MLA, LCA's design is architecture-agnostic and readily extends to other attention mechanisms such as GQA. Theoretically, we prove a length-independent error bound. Experiments show LCA achieves up to 2.5$\times$ prefilling speedup and 90% KV cache reduction at 128K context while maintaining competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。