提出恒定计算的Transformer架构，让长文本生成更快更省内存。
From TLinFormer to TConstFormer: The Leap to Constant-Time Transformer Attention: Achieving O(1) Computation and O(1) KV Cache during Autoregressive Inference
- 用周期性状态更新实现恒定大小的键值缓存
- 每256步仅1次全局同步，平均计算复杂度为O(1)
- 适合超长文本生成、实时流式语言模型应用
尽管Transformer已成为现代AI的核心，其自回归推理仍面临键值缓存线性增长和计算复杂度O(N²d)的问题，严重制约了对超长序列的处理能力。为此，本文提出TConstFormer架构，基于前期工作TLinFormer。TConstFormer采用创新的周期性状态更新机制，实现真正恒定大小的O(1) KV Cache。该机制在摊销意义上计算复杂度也为O(1)：连续执行k−1步（如k=256）为纯常数时间计算，仅在第k步进行一次线性时间的全局信息同步。理论分析与实验结果表明，TConstFormer在长文本推理任务中显著优于基线模型，在速度、内存效率与整体性能上均具压倒性优势。这一突破为高效可靠的流式语言模型应用铺平了道路。
原文摘要 · Abstract (English)
Although the Transformer has become the cornerstone of modern AI, its autoregressive inference suffers from a linearly growing KV Cache and a computational complexity of O(N^2 d), severely hindering its ability to process ultra-long sequences. To overcome this limitation, this paper introduces the TConstFormer architecture, building upon our previous work, TLinFormer. TConstFormer employs an innovative periodic state update mechanism to achieve a truly constant-size O(1) KV Cache. The computational complexity of this mechanism is also O(1) in an amortized sense: it performs purely constant-time computations for $k-1$ consecutive steps (e.g., $k=256$) and executes a single linear-time global information synchronization only on the $k$-th step. Theoretical calculations and experimental results demonstrate that TConstFormer exhibits an overwhelming advantage over baseline models in terms of speed, memory efficiency, and overall performance on long-text inference tasks. This breakthrough paves the way for efficient and robust streaming language model applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。