arXiv:2606.30355cs.CVcs.AI2026-06

利用缺失导致的特征偏差,让模型专家自动分工,提升不完整多模态预测效果。

Residual-Guided Expert Specialization for Incomplete Multimodal Learning

论文配图:Residual-Guided Expert Specialization for Incomplete Multimodal Learning
图 1 · 摘自论文原文
  • 通过对比完整与缺失输入的特征差异,生成残差信号指导专家分工。
  • 在多个数据集上优于基线,且在缺失场景下仍保持高效稳定。
  • 适合需要处理多模态缺失问题的实际系统部署,如跨模态识别与分割。

现实预测系统常面临推理时模态缺失的问题,不完整多模态学习(IML)仍是实际挑战。现有方法虽致力于学习对缺失输入鲁棒的表征,但缺失模态的表征不可避免地偏离完整模态表征。为此,我们提出MARS(缺失感知残差引导专业化)——一种基于残差信号的专家混合框架,根据表征如何被缺失性重塑来引导专家专业化。训练中,通过对比来自不完整输入与完整输入的任务表征,生成一个特权残差信号,捕捉表征差距;该信号引导残差路由器将样本分配给对应偏差模式的专家。同时,特征路由器仅使用不完整输入学习模仿此路由行为,实现无需完整模态即可部署。为缓解训练-测试路由差异,我们设计了一种差异感知噪声正则化,在特征路由器偏离时自适应扰动残差路由器决策,增强专家在不完美模仿下的鲁棒性。在多模态分类(CASIA-SURF、CREMA-D、UPMC Food-101)和分割(MCubeS)任务的缺失场景下实验表明,MARS持续超越基线,且具备高效率与可扩展性,适用于多种骨干网络与任务。

原文摘要 · Abstract (English)

As real-world prediction systems often face missing modalities at inference, incomplete multimodal learning (IML) remains a practical challenge. While prior methods aim to learn representations robust to missing inputs, representations from incomplete modalities inevitably deviate from their full-modality counterparts due to missing evidence. To explicitly leverage these deviations, we propose MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts framework that guides expert specialization based on how representations are reshaped by missingness. By contrasting task representations derived from incomplete inputs with their complete counterparts during training, we derive a privileged residual signal that captures this representational gap. The residual signal guides a residual router to assign samples to experts specialized for the corresponding deviation patterns. In parallel, a feature router learns to imitate this routing behavior using only incomplete inputs, enabling deployment without access to full modalities. To mitigate this train-test router gap, we develop a discrepancy-aware noise regularization that adaptively perturbs the residual router's decisions when the feature router deviates, enhancing expert robustness under imperfect imitation. Experiments on multimodal classification (CASIA-SURF, CREMA-D, UPMC Food-101) and segmentation (MCubeS) under missing scenarios show that MARS consistently surpasses baselines while remaining efficient and extensible to diverse backbones and tasks.

多模态学习缺失模态专家混合表征偏差

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。