arXiv:2604.20368cs.CVcs.AI2026-04

用拉普拉斯核替代softmax,提升视觉Transformer的效率与表达力。

LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel

论文配图:LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel
图 1 · 摘自论文原文
  • 以拉普拉斯核替代softmax,理论更扎实且保留中距离注意力。
  • 在ImageNet上实现高精度与低计算量平衡,优于现有线性注意力方法。
  • 专为边缘部署优化,支持高效前向与反向传播,适合实际应用。

Softmax注意力的二次复杂度阻碍了Transformer在高分辨率视觉任务中的扩展。现有线性注意力方法常以高斯核替代softmax,但此类近似缺乏理论基础,易抑制中等距离的标记交互。本文提出LaplacianFormer,采用拉普拉斯核作为软注意力的合理替代,基于经验观察与理论分析。为应对低秩近似下的表达力退化,引入可证明单射的特征映射,保留细粒度标记信息。计算方面,采用核矩阵的Nyström近似,并通过Newton-Schulz迭代求解,避免昂贵的矩阵求逆与SVD。进一步开发定制化CUDA实现,实现高吞吐的前向与反向传播,适用于边缘部署。ImageNet实验表明,LaplacianFormer在性能与效率间取得优异权衡,同时增强注意力表达能力。

原文摘要 · Abstract (English)

The quadratic complexity of softmax attention presents a major obstacle for scaling Transformers to high-resolution vision tasks. Existing linear attention variants often replace the softmax with Gaussian kernels to reduce complexity, but such approximations lack theoretical grounding and tend to oversuppress mid-range token interactions. We propose LaplacianFormer, a Transformer variant that employs a Laplacian kernel as a principled alternative to softmax, motivated by empirical observations and theoretical analysis. To address expressiveness degradation under low-rank approximations, we introduce a provably injective feature map that retains fine-grained token information. For efficient computation, we adopt a Nyström approximation of the kernel matrix and solve the resulting system using Newton--Schulz iteration, avoiding costly matrix inversion and SVD. We further develop custom CUDA implementations for both the kernel and solver, enabling high-throughput forward and backward passes suitable for edge deployment. Experiments on ImageNet show that LaplacianFormer achieves strong performance-efficiency trade-offs while improving attention expressiveness.

Transformer注意力机制线性注意力边缘部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。