Conformer编码器可能反转时间维度，影响语音识别效果。
The Conformer Encoder May Reverse the Time Dimension
- 自注意力机制使初始帧与后续信息建立连接，导致时间顺序反转。
- 训练中自注意力模块主导输出，仅允许反转信息通过。
- 提出避免反转的方法，并利用梯度获取标签位置对齐。
在基于Conformer的全局注意力编码器-解码器（AED）模型中，我们观察到交叉注意力权重单调递减。进一步分析发现，Conformer编码器在时间维度上反转了序列。研究解码器交叉注意力的初始行为，发现其促使编码器自注意力将初始帧与所有关键帧关联。此外，在训练过程中，自注意力模块逐渐主导输出，使前馈模块仅允许反转信息传递。本文提出避免此翻转的方法，并探索一种新方法：通过标签对数概率对编码器输入帧的梯度，获得标签-帧位置对齐。
原文摘要 · Abstract (English)
We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models, Further investigation shows that the Conformer encoder reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose methods and ideas of how this flipping can be avoided and investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。