解决多模态融合中噪声敏感和模态偏见问题,提升动态融合稳定性。
Unbiased Dynamic Multimodal Fusion
- 通过加噪反推机制,精准评估高低噪声下的模态不确定性。
- 引入模态依赖偏差量化,避免难学模态被双重压制。
- 在多个基准任务上超越静态与现有动态融合方法,适合复杂场景应用。
传统多模态方法假设模态质量恒定,难以适应真实世界中的动态变化。虽有动态方法可根据模态质量调整贡献,但通常依赖经验指标,在极低或极高噪声下表现不佳。此外,现有方法常假设各模态初始贡献相同,忽视了模态间的内在依赖偏见,导致难学模态被双重抑制,动态融合性能反而不如静态融合。为此,本文提出无偏动态多模态学习(UDML)框架:首先设计噪声感知的不确定性估计器,通过向模态数据添加可控噪声并从特征中预测噪声强度,建立特征退化与噪声水平的明确对应关系,实现对高低噪声条件下的准确不确定性度量;其次,利用模态丢弃法量化多模态网络中的内在模态依赖偏见,并将其融入权重机制,消除对难学模态的双重抑制。大量实验在多个多模态基准任务上验证了该方法的有效性、通用性与可推广性。代码已开源。
原文摘要 · Abstract (English)
Traditional multimodal methods often assume static modality quality, which limits their adaptability in dynamic real-world scenarios. Thus, dynamical multimodal methods are proposed to assess modality quality and adjust their contribution accordingly. However, they typically rely on empirical metrics, failing to measure the modality quality when noise levels are extremely low or high. Moreover, existing methods usually assume that the initial contribution of each modality is the same, neglecting the intrinsic modality dependency bias. As a result, the modality hard to learn would be doubly penalized, and the performance of dynamical fusion could be inferior to that of static fusion. To address these challenges, we propose the Unbiased Dynamic Multimodal Learning (UDML) framework. Specifically, we introduce a noise-aware uncertainty estimator that adds controlled noise to the modality data and predicts its intensity from the modality feature. This forces the model to learn a clear correspondence between feature corruption and noise level, allowing accurate uncertainty measure across both low- and high-noise conditions. Furthermore, we quantify the inherent modality reliance bias within multimodal networks via modality dropout and incorporate it into the weighting mechanism. This eliminates the dual suppression effect on the hard-to-learn modality. Extensive experiments across diverse multimodal benchmark tasks validate the effectiveness, versatility, and generalizability of the proposed UDML. The code is available at https://github.com/shicaiwei123/UDML.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。