线性注意力无法在实际模型宽度下进入核极限，影响了其可解释性可靠性。
Linearized Attention Cannot Enter the Kernel Regime at Any Practical Width
- 构建无参线性注意力与数据相关核的精确对应关系
- 需超大宽度（如10^29）才能收敛，远超现实模型规模
- 揭示注意力对数据扰动更敏感，适合研究模型脆弱性
理解注意力机制是否收敛至核极限，是确保变压器可解释性中影响函数有效性的基础。由于softmax注意力具有指数非线性，难以精确刻画其NTK；线性化注意力作为可处理的典型代理，是本文研究对象。本论文表明，即使该代理也无法在任何实际宽度下收敛至其NTK极限，揭示了注意力学习动态中的根本权衡。我们建立了无参线性注意力与数据依赖的格拉姆诱导核之间的精确对应；谱放大分析显示，注意力变换将格拉姆矩阵的条件数立方化，要求宽度 $m = Ω(κ_d( extbf{G})^6 n ext{log}n)$ 才能实现NTK收敛，其中 $κ_d( extbf{G})$ 是输入格拉姆矩阵秩为 $\min(n,d)$ 截断后的有效条件数；对于自然图像数据集，此阈值物理上不可行（MNIST 超 $10^{24}$，CIFAR-10 超 $10^{29}$，比现有最大架构高出12–17个数量级）。引入“影响可塑性”表征此不收敛现象：线性注意力在对抗数据扰动下表现出2–9倍于ReLU网络的可塑性，差距取决于数据集条件数和任务设定。理论与实证均表明，同一数据依赖核在目标与数据几何对齐时可降低近似误差，但同时导致训练数据易受对抗操纵。该结构性结论也适用于标准初始化下的可训练QKV注意力，对部署中变压器的影响方法有直接影响。
原文摘要 · Abstract (English)
Understanding whether attention mechanisms converge to the kernel regime is foundational to the validity of influence functions for transformer accountability. Exact NTK characterization of softmax attention is precluded by its exponential nonlinearity; linearized attention is the canonical tractable proxy and the object of study here. This paper establishes that even this proxy does not converge to its NTK limit at any practical width, revealing a fundamental trade-off in the learning dynamics of attention. An exact correspondence is established between parameter-free linearized attention and a data-dependent Gram-induced kernel; spectral amplification analysis shows that the attention transformation cubes the Gram matrix's condition number, requiring width $m = Ω(κ_d(\mathbf{G})^6 n\log n)$ for NTK convergence, where $κ_d(\mathbf{G})$ is the effective condition number of the rank-$\min(n,d)$ truncation of the input Gram matrix; for natural image datasets this threshold is physically infeasible ($m \gg 10^{24}$ for MNIST and $m \gg 10^{29}$ for CIFAR-10, 12--17 orders of magnitude beyond the largest known architectures). \emph{Influence malleability} is introduced to characterize this non-convergence: linearized attention exhibits 2--9$\times$ higher malleability than ReLU networks under adversarial data perturbation, with the gap depending on dataset condition number and task setting. A dual implication is established: the same data-dependent kernel is shown theoretically to reduce approximation error when targets align with the data geometry, while, empirically, creating vulnerability to adversarial manipulation of the training data. The structural argument extends to trainable QKV attention under standard initialization, with direct consequences for influence methods applied to deployed transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。