让多模态模型在学习初期更均衡地吸收信息，提升整体性能
Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition
- 在学习初期动态调节各模态信息获取速度，避免强势模态压制弱模态
- 在多个数据集上显著优于现有不平衡处理方法，性能全面提升
- 适合需要平衡多源信息输入的视觉-语言等多模态任务
早期感官训练对人类发展至关重要。受此启发，我们发现多模态学习中的早期阶段——即主学习窗口——同样关键，此时数据信息被快速吸收。然而，观察表明，该阶段常由信息丰富的模态主导，抑制了信息贫乏模态的学习。为此，我们提出信息获取调节（InfoReg），通过在主学习窗口减缓信息丰富模态的信息获取速度，促进信息贫乏模态的吸收。该调节使学习过程更均衡，提升了多模态网络的整体性能。实验表明，InfoReg在多个数据集上均优于现有不平衡处理方法，实现更优模型表现。代码已开源：https://github.com/GeWu-Lab/InfoReg_CVPR2025。
原文摘要 · Abstract (English)
Sensory training during the early ages is vital for human development. Inspired by this cognitive phenomenon, we observe that the early training stage is also important for the multimodal learning process, where dataset information is rapidly acquired. We refer to this stage as the prime learning window. However, based on our observation, this prime learning window in multimodal learning is often dominated by information-sufficient modalities, which in turn suppresses the information acquisition of information-insufficient modalities. To address this issue, we propose Information Acquisition Regulation (InfoReg), a method designed to balance information acquisition among modalities. Specifically, InfoReg slows down the information acquisition process of information-sufficient modalities during the prime learning window, which could promote information acquisition of information-insufficient modalities. This regulation enables a more balanced learning process and improves the overall performance of the multimodal network. Experiments show that InfoReg outperforms related multimodal imbalanced methods across various datasets, achieving superior model performance. The code is available at https://github.com/GeWu-Lab/InfoReg_CVPR2025.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。