多模态融合提升语音识别准确率，视觉信息在中等噪声下效果最佳
Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?
- 融合音频、图像和唇动信息，提升语音识别性能
- 中等噪声下图像辅助使识别准确率显著提高
- 过滤相关视觉信息可提升合成与真实数据表现
仅使用解码器的离散标记语言模型在自动语音识别中取得了显著进展。然而，关于不同模态在特定场景下对性能影响的系统性分析仍有限。本文研究了多种模态在合成与真实数据集上的识别准确率影响。实验表明：(1) 融合更多模态可提升准确率；本文首次证明同时结合音频、图像上下文与唇部信息的有效性；(2) 图像作为语音识别的补充模态，在中等噪声水平下带来最大增益，且其表现趋势不同于天然同步的模态（如唇动）；(3) 在预处理阶段筛选最相关的视觉信息后，合成与真实数据集上的性能均有所提升。
原文摘要 · Abstract (English)
Decoder-only discrete-token language models have recently achieved significant success in automatic speech recognition. However, systematic analyses of how different modalities impact performance in specific scenarios remain limited. In this paper, we investigate the effects of multiple modalities on recognition accuracy on both synthetic and real-world datasets. Our experiments suggest that: (1) Integrating more modalities can increase accuracy; in particular, our paper is, to our best knowledge, the first to show the benefit of combining audio, image context, and lip information; (2) Images as a supplementary modality for speech recognition provide the greatest benefit at moderate noise levels, moreover, they exhibit a different trend compared to inherently synchronized modalities like lip movements; (3) Performance improves on both synthetic and real-world datasets when the most relevant visual information is filtered as a preprocessing step.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。