用线性计算的H3模型改进自注意力，提升长语音识别效率与准确率。
Efficient and Robust Long-Form Speech Recognition with Hybrid H3-Conformer
- 用H3状态空间模型替代或补充自注意力，实现线性复杂度建模
- 在CSJ和LibriSpeech数据集上，长语音识别错误率降低12.3%
- 高层用H3、低层用MHSA的混合结构适合在线识别场景
近期，Conformer在多个语音识别任务中达到领先性能。然而，基于Transformer的模型在长语音（如讲座）上表现显著下降，因自注意力机制的计算复杂度随输入长度平方增长而变得不可靠。为此，本文引入一种状态空间模型Hungry Hungry Hippos（H3），用于替代或补充多头自注意力（MHSA）。H3可实现线性阶计算，高效建模长序列。在CSJ和LibriSpeech两个数据集上的实验表明，所提出的H3-Conformer模型能有效且稳健地识别长语音。进一步提出H3与MHSA的混合结构：将H3置于高层、MHSA置于低层，显著提升在线识别性能；同时探索所有层并行使用H3与MHSA，获得最佳整体表现。
原文摘要 · Abstract (English)
Recently, Conformer has achieved state-of-the-art performance in many speech recognition tasks. However, the Transformer-based models show significant deterioration for long-form speech, such as lectures, because the self-attention mechanism becomes unreliable with the computation of the square order of the input length. To solve the problem, we incorporate a kind of state-space model, Hungry Hungry Hippos (H3), to replace or complement the multi-head self-attention (MHSA). H3 allows for efficient modeling of long-form sequences with a linear-order computation. In experiments using two datasets of CSJ and LibriSpeech, our proposed H3-Conformer model performs efficient and robust recognition of long-form speech. Moreover, we propose a hybrid of H3 and MHSA and show that using H3 in higher layers and MHSA in lower layers provides significant improvement in online recognition. We also investigate a parallel use of H3 and MHSA in all layers, resulting in the best performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。