提出分层核注意力机制，高效实现多尺度序列建模。
Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis
- 通过可学习的因果下采样在多分辨率层级上处理序列，融合各层级注意力得分。
- 在3个层级下仅增加1.31倍计算开销，实验中各项任务提升4.77至7.47个百分点。
- 理论完备，兼具对称与反对称注意力分解，适合需要高效长序列建模的研究者。
分层核注意力（HKT）是一种多尺度注意力机制，通过可学习的因果下采样在L个分辨率层级上处理序列，并通过可学习的凸权重组合各层级的得分矩阵。总计算成本上限为标准注意力的4/3倍，当L=3时为1.3125倍。建立了四项理论结果：(i) 在对称双线性形式满足充分条件时，分层得分矩阵定义了一个半正定核（命题3.1）；(ii) 非对称得分矩阵可唯一分解为控制相互注意力的对称部分和控制方向注意力的反对称部分；HKT在每个层级提供独立的一对，共L组（命题3.5-3.6）；(iii) 近似误差可分解为三个可解释成分，包含显式非高斯修正项，并在层级数L上具有几何衰减界（定理4.3，命题4.4）；(iv) HKT严格包含单头标准注意力与因果卷积（命题3.4）。在3个随机种子下的实验显示，相较于重训练的标准注意力基线，一致取得性能提升：合成ListOps任务+4.77个百分点（55.10±0.29% vs 50.33±0.12%，T=512），序列CIFAR-10任务+1.44个百分点（35.45±0.09% vs 34.01±0.19%，T=1,024），IMDB字符级情感分析任务+7.47个百分点（70.19±0.57% vs 62.72±0.40%，T=1,024），所有实验均在1.31倍开销内完成。
原文摘要 · Abstract (English)
The Hierarchical Kernel Transformer (HKT) is a multi-scale attention mechanism that processes sequences at L resolution levels via trainable causal downsampling, combining level-specific score matrices through learned convex weights. The total computational cost is bounded by 4/3 times that of standard attention, reaching 1.3125x for L = 3. Four theoretical results are established. (i) The hierarchical score matrix defines a positive semidefinite kernel under a sufficient condition on the symmetrised bilinear form (Proposition 3.1). (ii) The asymmetric score matrix decomposes uniquely into a symmetric part controlling reciprocal attention and an antisymmetric part controlling directional attention; HKT provides L independent such pairs across scales, one per resolution level (Propositions 3.5-3.6). (iii) The approximation error decomposes into three interpretable components with an explicit non-Gaussian correction and a geometric decay bound in L (Theorem 4.3, Proposition 4.4). (iv) HKT strictly subsumes single-head standard attention and causal convolution (Proposition 3.4). Experiments over 3 random seeds show consistent gains over retrained standard attention baselines: +4.77pp on synthetic ListOps (55.10+-0.29% vs 50.33+-0.12%, T = 512), +1.44pp on sequential CIFAR-10 (35.45+-0.09% vs 34.01+-0.19%, T = 1,024), and +7.47pp on IMDB character-level sentiment (70.19+-0.57% vs 62.72+-0.40%, T = 1,024), all at 1.31x overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。