用检索代替生成,恢复缺失音视频信息以提升问答准确率
Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

- 通过跨模态检索获取缺失模态的语义信息
- 引入自适应净化机制消除检索结果中的噪声
- 适合处理真实场景中数据不全的音视频问答任务
近年来音频-视觉问答(AVQA)方法取得显著进展。然而,大多数方法缺乏有效处理缺失模态的机制,在实际场景中因数据中断导致性能严重下降。现有方法主要依赖生成式补全来合成缺失特征,虽部分有效,但往往只捕捉跨模态共性,难以保留缺失数据中的模态特异性知识,引发幻觉并降低推理准确性。为此,我们提出R²ScP框架,将缺失模态处理范式从传统生成补全转向基于检索的恢复。具体而言,利用统一语义嵌入进行跨模态检索,获取缺失领域的特定知识;为最大化语义还原,设计上下文感知的自适应净化机制,剔除检索数据中的潜在语义噪声;此外,采用两阶段训练策略显式建模不同来源知识间的语义关系。大量实验表明,R²ScP显著提升了AVQA性能,并增强了在模态不完整场景下的鲁棒性。
原文摘要 · Abstract (English)
Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly. However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions. Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features. While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modality-specific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy. To tackle these challenges, we propose R$^{2}$ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery. Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge. To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data. Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources. Extensive experiments demonstrate that R$^{2}$ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。