动态调整多模态模型中各模态贡献,提升噪声环境下的性能
Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency
- 根据置信度、不确定性与语义一致性动态分配模态权重
- 在视觉问答等任务上显著提升干净与受损数据表现
- 适用于现有模型,尤其适合模态缺失或错位场景
多模态大模型在视觉-语言理解与生成任务中取得显著进展,但现有方法通常采用静态模态融合策略,对所有模态一视同仁,忽略其在实例级别上的可靠性与语义贡献。这导致在存在噪声、缺失或错位模态时性能下降。本文提出动态模态调度(DMS),一种在样本级自适应调整各模态贡献的新框架。DMS基于三个关键因素评估模态:(1) 置信度,由预测熵估计;(2) 不确定性,通过蒙特卡洛丢弃获得;(3) 语义一致性,通过跨模态相似性计算。这些信号通过可学习或规则化的调度器整合为软模态权重,用于下游融合。为保证训练稳定,引入模态权重一致性损失,使融合表示与单模态嵌入保持比例一致。该方法具有模型无关性,可集成至BLIP-2、LLaVA等模型。在视觉问答、图像-文本检索和图像描述任务上的实验表明,DMS显著提升清洁数据与鲁棒性表现,尤其在模态被破坏或丢弃条件下效果更佳。本工作提供了一种通用且有效的实例感知多模态建模机制。
原文摘要 · Abstract (English)
Multimodal Large Models (MLLMs) have achieved remarkable progress in vision-language understanding and generation tasks. However, existing MLLMs typically rely on static modality fusion strategies, which treat all modalities equally regardless of their instance-level reliability or semantic contribution. This often leads to suboptimal performance, especially in scenarios with noisy, missing, or misaligned modalities. In this paper, we propose Dynamic Modality Scheduling (DMS), a novel framework that adaptively adjusts the contribution of each modality at a per-sample level. DMS evaluates each modality based on three key factors: (1) \textit{confidence}, estimated from predictive entropy; (2) \textit{uncertainty}, obtained via Monte Carlo dropout; and (3) \textit{semantic consistency}, computed through inter-modal similarity. These signals are combined through a learnable or rule-based scheduler to generate soft modality weights used in downstream fusion.To ensure stable training, we further introduce a \textit{Modality Weight Consistency Loss}, which regularizes the fused representation to stay close to unimodal embeddings proportionally to their assigned weights. Our method is model-agnostic and can be integrated into existing MLLMs such as BLIP-2 and LLaVA. Experimental results on VQA, image-text retrieval, and captioning tasks show that DMS significantly improves both clean and robust performance, especially under modality corruption or dropout conditions. This work provides a general and effective mechanism to enable instance-aware and robustness-enhanced multimodal modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。