用信息论分析视听任务中的信息重叠，揭示融合优势与难点
Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective
- 基于信息论量化多模态间的信息交集
- 发现模态融合可提升识别性能，尤其在低信噪比下
- 为视听语音识别提供理论依据，适合做多模态研究者
在语音处理领域，视听语音处理正受到越来越多关注。核心任务包括唇读、视听语音识别和视觉到语音合成。尽管已取得显著进展，但对视听任务的理论分析仍显不足。本文基于信息论进行定量分析，聚焦不同模态间的信息交集。结果表明，该分析有助于理解视听处理任务的挑战以及模态融合带来的潜在收益。
原文摘要 · Abstract (English)
In the field of spoken language processing, audio-visual speech processing is receiving increasing research attention. Key components of this research include tasks such as lip reading, audio-visual speech recognition, and visual-to-speech synthesis. Although significant success has been achieved, theoretical analysis is still insufficient for audio-visual tasks. This paper presents a quantitative analysis based on information theory, focusing on information intersection between different modalities. Our results show that this analysis is valuable for understanding the difficulties of audio-visual processing tasks as well as the benefits that could be obtained by modality integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。