揭示多模态融合中模态崩溃的根源并提出缓解方法
A Closer Look at Multimodal Representation Collapse
- 发现噪声特征通过共享神经元纠缠导致模态崩溃
- 跨模态知识蒸馏可解耦表示，提升融合效果
- 提出显式基底重分配算法，支持缺失模态场景
我们旨在深入理解一种近期观察到的多模态融合现象——模态崩溃，即模型在训练过程中仅依赖部分模态而忽略其余模态。研究发现，当某一模态的噪声特征通过融合头中的共享神经元与另一模态的预测特征纠缠时，会掩盖前一模态的正向贡献，导致其崩溃。进一步证明，跨模态知识蒸馏可通过释放学生编码器中的秩瓶颈，隐式解耦表示，实现去噪且不损害任一模态的预测特征。基于此，我们提出一种通过显式基底重分配防止模态崩溃的算法，适用于处理缺失模态问题。在多个多模态基准上的大量实验验证了理论结论。
原文摘要 · Abstract (English)
We aim to develop a fundamental understanding of modality collapse, a recently observed empirical phenomenon wherein models trained for multimodal fusion tend to rely only on a subset of the modalities, ignoring the rest. We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with predictive features from another, effectively masking out positive contributions from the predictive features of the former modality and leading to its collapse. We further prove that cross-modal knowledge distillation implicitly disentangles such representations by freeing up rank bottlenecks in the student encoder, denoising the fusion-head outputs without negatively impacting the predictive features from either modality. Based on the above findings, we propose an algorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities. Extensive experiments on multiple multimodal benchmarks validate our theoretical claims. Project page: https://abhrac.github.io/mmcollapse/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。