用沙普利值解析音视频语音识别中模态贡献，发现模型始终偏音频。
Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
- 基于沙普利值分析音视频模态在识别中的动态贡献
- 噪声下模型更依赖视觉，但音频贡献仍很高
- 适合研究音视频融合机制的开发者和评估者
音视频语音识别（AVSR）利用声学与视觉信息提升噪声环境下的识别鲁棒性。然而，模型如何权衡两种模态仍不清晰。本文提出 Dr. SHAP-AV 框架，采用沙普利值分析 AVSR 中的模态贡献。在六个模型、两个基准数据集及不同信噪比（SNR）条件下进行实验，引入三种分析：全局沙普利值（Global SHAP）用于整体模态平衡，生成沙普利值（Generative SHAP）揭示解码过程中的贡献动态，时序对齐沙普利值（Temporal Alignment SHAP）检验输入输出对应关系。结果表明，噪声环境下模型倾向依赖视觉，但音频贡献依然显著；模态平衡随生成过程演变，时序对齐在噪声下仍保持稳定，且信噪比是主导模态权重的关键因素。研究揭示了持续存在的音频偏倚，推动即兴模态加权机制的发展，并倡导沙普利值归因作为标准的 AVSR 分析工具。
原文摘要 · Abstract (English)
Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。