解决语音模型在长音频上注意力失效问题，让模型能准确对齐长段落语音。
Segmental Attention Decoding With Long Form Acoustic Encodings
- 为每个解码段显式注入绝对位置编码，克服注意力机制的顺序盲区。
- 通过长序列训练和段落拼接，使模型在无边界提示时仍保持定位能力。
- 适合需要高精度长语音识别的场景，如会议转录、播客分析。
我们解决基于注意力的编码器-解码器（AED）模型与长语音编码不兼容的根本问题。在分段语音上训练的AED模型会依赖段边界外的有限声学上下文来学习绝对帧位置，但在解码长语音时这些线索消失，导致模型因交叉注意力中键值的排列不变性而失去对声学编码的排序能力。为此我们提出四项改进：(1) 在每个解码段的交叉注意力中注入显式的绝对位置编码；(2) 使用扩展声学上下文进行长语音训练，消除隐式位置编码；(3) 段落拼接以覆盖训练所需的多样化分段方式；(4) 语义分段以对齐AED解码段与训练段。实验表明这些方法弥合了连续与分段声学编码间的性能差距，实现了注意力解码器的自回归使用。
原文摘要 · Abstract (English)
We address the fundamental incompatibility of attention-based encoder-decoder (AED) models with long-form acoustic encodings. AED models trained on segmented utterances learn to encode absolute frame positions by exploiting limited acoustic context beyond segment boundaries, but fail to generalize when decoding long-form segments where these cues vanish. The model loses ability to order acoustic encodings due to permutation invariance of keys and values in cross-attention. We propose four modifications: (1) injecting explicit absolute positional encodings into cross-attention for each decoded segment, (2) long-form training with extended acoustic context to eliminate implicit absolute position encoding, (3) segment concatenation to cover diverse segmentations needed during training, and (4) semantic segmentation to align AED-decoded segments with training segments. We show these modifications close the accuracy gap between continuous and segmented acoustic encodings, enabling auto-regressive use of the attention decoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。