arXiv:2607.05019cs.LGcs.CV2026-07

不融合模态也能更好分类,用多个单模态模型组合提升性能

Beyond Modality Fusion: Deep Ensembles for Multimodal Classification

论文配图:Beyond Modality Fusion: Deep Ensembles for Multimodal Classification
图 1 · 摘自论文原文
  • 用多个单模态网络的深度集成替代传统模态融合方法
  • 在参数量相当情况下,集成方法持续优于先进融合模型
  • 适合处理模态不平衡问题,尤其小规模集成时更有效

在多模态分类中,晚期融合方法通过拼接各模态独立神经网络提取的特征进行分类。当模态间存在显著不平衡时,已有多种正则化技术用于平衡学习过程。本文表明,无需显式模态融合,仅通过单模态网络的深度集成即可高效分类多模态数据。我们在相同参数量下系统比较了集成方法与晚期融合网络,结果表明集成方法始终优于针对模态不平衡设计的先进融合方法。该优势同样超过中间融合和混合预测方法。我们提出并验证了一种选择每模态模型数量的方法,避免耗时的穷举搜索。在极端模态不平衡且集成规模较小时,仅使用强模态训练的单模态模型集成更优;随着集成规模增大,引入弱模态模型变得有利,与实验结果一致。为系统研究多模态模型优化挑战,我们构建了一个可调控模态数量和预测能力的合成框架,发现结果在合成与真实数据集上均一致。最后,通过对双模态数据集拟合缩放定律,估计了集成方法的渐近性能。

原文摘要 · Abstract (English)

In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks. When modality imbalance is pronounced, various regularization techniques have been proposed to balance the learning process and overcome the inferior performance of late-fusion networks. In contrast, this work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. We systematically compare deep ensembles to late-fusion networks at equal parameter count and show that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance. This advantage also holds over intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions. We propose and empirically validate a method for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance and small ensemble sizes, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable; as the ensemble scales up, incorporating models from the weaker modality becomes beneficial. Both predictions align with our empirical findings. To systematically explore the challenges of optimizing multimodal models, we propose a synthetic multimodal framework that allows control over both the number of modalities and their predictive strength; our findings are consistent across synthetic and real-world datasets. Finally, by fitting scaling laws to bimodal datasets, we estimate the asymptotic performance of ensembles.

多模态深度集成分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。