通过自适应调节冗余,平衡多模态训练中的信息优化。
Adaptive Redundancy Regulation for Balanced Multimodal Information Refinement
- 基于信息瓶颈原理,动态监测冗余并触发干预。
- 在多数任务中优于现有方法,提升多模态性能。
- 适合需要平衡多源数据的模型训练场景。
多模态学习旨在通过融合多源数据提升性能。联合训练中,由于模态偏差,优势模态常主导反向传播,导致优化不平衡。现有方法存在两大问题:其一,优势模态长期主导削弱了表示与输出间的耦合,造成冗余信息累积;其二,以往方法常直接均匀调整优势模态梯度,忽视模态间语义与方向性。为此,本文提出自适应冗余调节机制(RedReg),受信息瓶颈原理启发。具体地,构建冗余阶段监测器,结合有效增益增长率与冗余度,在冗余较高时触发干预。设计共信息门控机制,基于跨模态语义估计当前优势模态贡献。当任务主要依赖单一模态时,抑制项自动关闭以保留特异性信息。最后,将优势模态梯度投影到联合多模态梯度子空间的正交补空间,并按冗余程度抑制梯度。实验表明,该方法在多数场景下优于当前主流方法。消融实验验证了有效性。代码已开源:https://github.com/xia-zhe/RedReg.git。
原文摘要 · Abstract (English)
Multimodal learning aims to improve performance by leveraging data from multiple sources. During joint multimodal training, due to modality bias, the advantaged modality often dominates backpropagation, leading to imbalanced optimization. Existing methods still face two problems: First, the long-term dominance of the dominant modality weakens representation-output coupling in the late stages of training, resulting in the accumulation of redundant information. Second, previous methods often directly and uniformly adjust the gradients of the advantaged modality, ignoring the semantics and directionality between modalities. To address these limitations, we propose Adaptive Redundancy Regulation for Balanced Multimodal Information Refinement (RedReg), which is inspired by information bottleneck principle. Specifically, we construct a redundancy phase monitor that uses a joint criterion of effective gain growth rate and redundancy to trigger intervention only when redundancy is high. Furthermore, we design a co-information gating mechanism to estimate the contribution of the current dominant modality based on cross-modal semantics. When the task primarily relies on a single modality, the suppression term is automatically disabled to preserve modality-specific information. Finally, we project the gradient of the dominant modality onto the orthogonal complement of the joint multimodal gradient subspace and suppress the gradient according to redundancy. Experiments show that our method demonstrates superiority among current major methods in most scenarios. Ablation experiments verify the effectiveness of our method. The code is available at https://github.com/xia-zhe/RedReg.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。