发现注意力稀疏是可学习的拓扑结构，实现无损加速。
The Condensate Theorem: Transformers are O(n), Not $O(n^2)$
- 通过动态识别注意力集中流形，实现精确替代全量注意力
- 在多个模型上实现159倍加速，131K token时仅需3.94ms
- 适合追求极致推理效率的部署场景
我们提出凝聚态定理：注意力稀疏是可学习的拓扑特性，而非架构限制。通过对训练好的语言模型进行实证分析，发现注意力质量集中在特定拓扑流形上——该流形可动态识别而无需检查所有位置。我们证明了一个通用结论：对任意查询，将注意力投影到凝聚态流形（锚点+窗口+动态Top-k）能与全量O(n²)注意力实现100%输出等价。这并非近似，而是无损一致性。我们在GPT-2、Pythia、Qwen2、TinyLlama和Mistral上验证，生成超过1,500个令牌时实现比特级完全匹配。将此拓扑映射至硬件后，拓扑注意力内核在131K令牌下实现159倍实测加速（3.94ms vs 628ms），在100万令牌下预估超1,200倍加速，相比Flash Attention推理成本降低超过99.9%。结论：二次复杂度瓶颈是朴素实现的结果，而非智能本质。
原文摘要 · Abstract (English)
We present the Condensate Theorem: attention sparsity is a learned topological property, not an architectural constraint. Through empirical analysis of trained language models, we find that attention mass concentrates on a distinct topological manifold -- and this manifold can be identified dynamically without checking every position. We prove a general result: for any query, projecting attention onto the Condensate Manifold (Anchor + Window + Dynamic Top-k) achieves 100% output equivalence with full $O(n^2)$ attention. This is not an approximation -- it is lossless parity. We validate this across GPT-2, Pythia, Qwen2, TinyLlama, and Mistral, demonstrating bit-exact token matching on 1,500+ generated tokens. By mapping this topology to hardware, our Topological Attention kernel achieves a 159x measured speedup at 131K tokens (3.94ms vs 628ms) and a projected >1,200x speedup at 1M tokens, reducing inference costs by >99.9% compared to Flash Attention. We conclude that the quadratic bottleneck is an artifact of naive implementation, not intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。