arXiv:2605.26232cs.CV2026-05

让视频模型按指令智能选择有用模态,避免无关信息干扰。

Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos

论文配图:Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
图 1 · 摘自论文原文
  • 根据文本指令动态调节各模态重要性,分层控制信息融合。
  • 在6个基准上最高提升13.5的CIDEr指标,优于固定融合方法。
  • 适合多模态视频理解任务,尤其处理音频、深度图等复杂输入。

预训练视频大语言模型在视觉推理上表现优异,但在引入音频、深度图或密集时间证据等辅助流时表现下降。统一融合导致模态干扰,无关通道分散模型注意力。为此,我们提出统一的多模态视频理解框架UniMVU,通过两层动态门控实现指令感知融合:模态内门控强调每种模态中的显著区域,模态级门控重新加权完整数据流;两者均基于文本指令动态调整。UniMVU结合跨模态自注意力与指令驱动的模态内门控模块及带控制令牌的模态级门控模块,并对时序对齐数据采用快到慢融合策略以减少冗余。在六个基准(AVQA、AVSD、Music-AVQA、ScanQA、SQA3D和MVBench)上,相比静态融合基线,UniMVU取得一致提升,最高达13.5的CIDEr得分。分析显示,门控机制与人类可解释的模态相关性一致,消融实验验证了模态内与模态级门控的贡献。UniMVU为多模态视频理解提供简单统一的指令感知方案,可扩展至多种模态而无需手工设计融合规则。

原文摘要 · Abstract (English)

Pre-trained video large language models excel at visual reasoning. However, they struggle when videos arrive with auxiliary streams, such as audio, depth map, or dense temporal evidence. In such a scenario, uniform fusion induces modality interference, allowing irrelevant channels to distract the model. To address this issue, we present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs via two levels of dynamic gating: inner-modality gates emphasize salient regions within each modality, whereas modality-level gates re-weight whole streams; both are conditioned on the text instruction to adaptively balance modality importance. Our UniMVU combines cross-modal self-attention with instruction-driven inner-modality gating module and a modality-level gating module with control token; for time-aligned streams we further adopt a fast-to-slow fusion scheme that reduces redundancy. Across six benchmarks (AVQA, AVSD, Music-AVQA, ScanQA, SQA3D and MVBench), our UniMVU achieves consistent gains over static-fusion baselines achieving gains as high as 13.5 in terms of CIDEr metric. Further, our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating. Our UniMVU provides a simple, unified recipe for instruction-aware multimodal video understanding that scales to diverse modalities without hand-crafted fusion rules.

多模态视频理解门控机制指令感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。