揭示张量注意力中键值缓存压缩的时空权衡机制
Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding
- 通过通信复杂性归约，推导出张量注意力的内存下界
- 在 d = Ω(log n) 条件下，给出两种张量缓存方案的时空权衡
- 为高效张量注意力模型设计提供理论依据，适合关注推理优化的研究者
张量版本的 Transformer 中的键值（KV）缓存是推理阶段的重大瓶颈。尽管已有研究分析了标准注意力机制中的空间复杂度极限 [Haris and Onak, 2025]，本文将该空间复杂度下界结果推广至张量注意力机制。理论分析基于通信复杂性归约，推导出当 d = Ω(log n) 时，张量结构注意力机制的内存下界。此外，我们提出两类张量注意力缓存，并针对两种场景建立了时间与内存之间的权衡关系。整体工作为理解张量注意力解码中 KV 缓存压缩的时空权衡提供了理论基础，也为开发更高效的内存型张量注意力 Transformer 架构提供了新视角。
原文摘要 · Abstract (English)
The key-value (KV) cache in the tensor version of transformers presents a significant bottleneck during inference. While previous work analyzes the fundamental space complexity barriers in standard attention mechanisms [Haris and Onak, 2025], our work generalizes the space complexity barriers result to tensor attention version. Our theoretical contributions rely on a reduction from communication complexity and deduce the memory lower bound for tensor-structured attention mechanisms when $d = Ω(\log n)$. Furthermore, we introduce two types of tensor attention cache and present a trade-off between time and memory for two scenarios. Overall, our work provides a theoretical foundation for us to understand the time-memory tradeoff of KV-Cache compression in tensor attention decoding and offers more perspectives in developing more memory-efficient tensor attention Transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。