提出TLinFormer,用新连接结构实现线性复杂度的精确注意力。
Rethinking Transformer Connectivity: TLinFormer, A Path to Exact, Full Context-Aware Linear Attention
- 重构神经元连接模式,实现严格线性复杂度
- 在长序列任务中推理延迟降低、显存占用更少
- 适合需要高效处理长序列的应用场景
Transformer架构已成为现代人工智能的核心,但其自注意力机制存在随序列长度呈二次增长的复杂度瓶颈,严重限制了在长序列任务中的应用。现有线性注意力方法通常通过数据无关的核近似或受限的上下文选择牺牲模型性能。本文回归连接主义本质,从信息流拓扑结构出发,提出新型线性注意力架构——TLinFormer。通过重新配置神经元连接模式,TLinFormer在保持严格线性复杂度的同时计算精确注意力分数,并确保信息流始终感知完整历史上下文。该设计旨在弥合现有高效注意力方法与标准注意力之间的性能差距。通过一系列实验,系统评估了TLinFormer在长序列推理任务中相对于标准Transformer基线的表现。结果表明,TLinFormer在推理延迟、KV缓存效率、内存占用和整体加速比等关键指标上均展现出显著优势。
原文摘要 · Abstract (English)
The Transformer architecture has become a cornerstone of modern artificial intelligence, but its core self-attention mechanism suffers from a complexity bottleneck that scales quadratically with sequence length, severely limiting its application in long-sequence tasks. To address this challenge, existing linear attention methods typically sacrifice model performance by relying on data-agnostic kernel approximations or restrictive context selection. This paper returns to the first principles of connectionism, starting from the topological structure of information flow, to introduce a novel linear attention architecture-\textbf{TLinFormer}. By reconfiguring neuron connection patterns, TLinFormer achieves strict linear complexity while computing exact attention scores and ensuring information flow remains aware of the full historical context. This design aims to bridge the performance gap prevalent between existing efficient attention methods and standard attention. Through a series of experiments, we systematically evaluate the performance of TLinFormer against a standard Transformer baseline on long-sequence inference tasks. The results demonstrate that TLinFormer exhibits overwhelming advantages in key metrics such as \textbf{inference latency}, \textbf{KV cache efficiency}, \textbf{memory footprint}, and \textbf{overall speedup}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。