用信息分解动态调整多模态训练顺序,提升模型学习效率
SPICE: Synergy and Partial Information Based Curriculum Evolution

- 基于信息分解理论拆解多模态信息为冗余、独有和协同三类
- 训练中实时优化样本顺序,逐步引导模型学习共享、特异到复杂协同信息
- 在多个基准上超越传统方法,适合需要高效多模态训练的研究者
多模态学习利用异构模态间的互补信息。各样本及训练阶段中,每种模态的信息量差异显著。现有方法通常假设样本相对复杂度不变,难以适应模型演进。本文提出SPICE(基于协同与部分信息的课程演化框架),一种新颖的多模态交互学习渐进式课程框架。受部分信息分解(PID)理论指导,该方法将多模态交互分解为冗余、独特和协同信息成分,实现可解释且动态的样本复杂度刻画。基于此分解,设计随训练动态演化的课程,使模型能从学习跨模态共享线索,过渡到模态特异性模式,最终掌握复杂协同交互。通过单模态与多模态预测所得的PID信息估计,实时优化样本排序。在多个多模态基准上的实验表明,其性能持续优于常规训练和最先进基线,验证了PID信息分解与自适应样本排序的有效性。
原文摘要 · Abstract (English)
Multimodal learning exploits complementary information across heterogeneous modalities. The informativeness of each modality can vary widely across samples and training stages. Existing multimodal curriculum learning strategies often assume that the relative complexity of samples remains unchanged throughout training and therefore cannot adapt to model evolution. We propose SPICE (Synergy and Partial Information based Curriculum Evolution), a novel progressive curriculum framework for multimodal interaction learning. Guided by Partial Information Decomposition (PID) theory, our approach decomposes multimodal interactions into redundant, unique, and synergistic information components, enabling an interpretable and dynamic characterization of sample complexity. Building on this decomposition, we design a progressive curriculum that evolves throughout training, allowing the model to transition from learning shared cross-modal cues to modality-specific patterns and, finally, to complex synergistic interactions. Adapting to model evolution, sample ordering is refined in real-time using PID information estimates derived from unimodal and multimodal predictions. Experiments across multiple multimodal benchmarks demonstrate consistent improvements over conventional training and state-of-the-art baselines, highlighting the effectiveness of PID information decomposition and adaptive sample ordering for multimodal curriculum learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。