arXiv:2603.02695cs.LG2026-03被引 1

统一处理模态缺失与噪声,提升低质量多模态数据的模型鲁棒性。

Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data

  • 将缺失和噪声模态视为统一的低质量问题,设计联合解决方案。
  • 在多个数据集上,完整、缺失、噪声场景下均超越现有最优方法。
  • 适合多模态情感计算中存在数据不全或干扰的场景。

真实场景中的多模态数据通常质量低下,噪声模态和缺失模态是常见问题,严重制约模型性能与鲁棒性。以往研究多分别处理噪声与缺失模态,而本文提出统一模态质量(UMQ)框架,联合应对两类问题,以增强多模态情感计算中的低质量表征能力。首先,通过排名引导训练策略,引入相对质量约束,构建带显式监督信号的质量评估器,避免因绝对质量标签不准带来的训练噪声。其次,为每类模态设计质量增强模块,利用其他模态提供的样本特异性信息及定义的模态基准表示,优化单模态表征质量。最后,提出一种质量感知的专家混合模块,结合特定路由机制,更精准地处理多种模态质量问题。UMQ 在多个数据集上,在完整、缺失、噪声三种设置下均持续优于现有最优基线方法。

原文摘要 · Abstract (English)

Multimodal data encountered in real-world scenarios are typically of low quality, with noisy modalities and missing modalities being typical forms that severely hinder model performance and robustness. However, prior works often handle noisy and missing modalities separately. In contrast, we jointly address missing and noisy modalities to enhance model robustness in low-quality data scenarios. We regard both noisy and missing modalities as a unified low-quality modality problem, and propose a unified modality-quality (UMQ) framework to enhance low-quality representations for multimodal affective computing. Firstly, we train a quality estimator with explicit supervised signals via a rank-guided training strategy that compares the relative quality of different representations by adding a ranking constraint, avoiding training noise caused by inaccurate absolute quality labels. Then, a quality enhancer for each modality is constructed, which uses the sample-specific information provided by other modalities and the modality-specific information provided by the defined modality baseline representation to enhance the quality of unimodal representations. Finally, we propose a quality-aware mixture-of-experts module with particular routing mechanism to enable multiple modality-quality problems to be addressed more specifically. UMQ consistently outperforms state-of-the-art baselines on multiple datasets under the settings of complete, missing, and noisy modalities.

多模态质量增强缺失模态噪声处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。