用可学习脉冲替代注意力，让语音模型在手机上更快更省电。
Learnable Pulse Accumulation for On-Device Speech Recognition: How Much Attention Do You Need?
- 用可学习的脉冲函数替代自注意力中的点积运算，复杂度从O(n²)降为O(n)。
- 在120秒音频上实现3.27倍加速，词错误率降低7.24个百分点至10.61%。
- 适合边缘设备部署，支持移动端神经加速器，无需频繁同步处理器。
自注意力机制随序列长度呈二次方增长，限制了基于Transformer的语音模型在边缘设备上的应用。本文提出可学习脉冲累积器（LPA），一种O(n)的替代方案，将键-查询点积替换为可学习的门控函数：内容相关的矩形脉冲、周期性窗口和位置相关的基函数。通过均方误差诊断扫描确定各层替换难度与顺序。替换wav2vec2-base模型中12层中的8层后，在LibriSpeech test-clean上达到10.61%的词错误率（WER），相比3.37%的基线提升7.24个百分点，且在Apple M4 Pro上实现120秒音频3.27倍加速，得益于优化的MLX推理路径。跨领域验证显示，SepFormer语音增强模型中所有16个块内注意力层均可被替换而不崩溃，表明深度瓶颈源于语言计算而非LPA本身局限。推理时近二值化的门控结构支持密集GPU计算，无须CPU-GPU同步，所有操作均可映射至移动神经加速器。
原文摘要 · Abstract (English)
Self-attention scales quadratically with sequence length, limiting transformer-based speech models on edge devices. We introduce the Learnable Pulse Accumulator (LPA), an O(n) replacement that substitutes key-query dot products with learned gating functions: content-dependent rectangular pulses, periodic windows, and position-dependent basis functions. An MSE diagnostic sweep determines per-layer replacement difficulty and ordering. Replacing 8 of 12 wav2vec2-base layers yields 10.61% word error rate (WER) on LibriSpeech test-clean, +7.24 percentage points (pp) over the 3.37% baseline, with 3.27x speedup at 120s audio on Apple M4 Pro via an optimized MLX inference path. Cross-domain validation on SepFormer speech enhancement shows all 16 intra-chunk attention layers can be replaced without collapse, suggesting the depth wall arises from linguistic computation rather than an LPA limitation. LPA's near-binary gates at inference enable dense GPU computation with no CPU-GPU synchronization, and all operations map to mobile neural accelerators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。