用博弈论正则化平衡多模态训练,避免模态竞争导致的性能下降。
Balancing Multimodal Training Through Game-Theoretic Regularization
- 基于互信息分解设计博弈框架,动态调节各模态贡献。
- 提升任务相关共享与独有信息的提取效率,改善联合训练效果。
- 通过潜在空间置换加速条件互信息计算,适合大规模多模态应用。
多模态学习通过捕捉跨数据源的依赖关系,有望实现更丰富的信息提取。然而,当前训练方法常因模态竞争而表现不佳,即各模态争夺训练资源导致部分模态优化不足。本文提出多模态竞争正则化器(MCR),受互信息(MI)分解启发,旨在缓解多模态训练中的负面竞争效应。核心贡献包括:1)构建博弈论框架,自适应平衡模态贡献,促使每个模态最大化其对最终预测的信息价值;2)精炼每项互信息的上下界,增强任务相关共享与独有信息的提取能力;3)提出潜在空间置换方法用于条件互信息估计,显著提升计算效率。MCR在合成数据和真实大型数据集上均优于所有先前训练策略及基础模型,明确验证了联合训练模态带来的显著性能提升。代码与模型已开源于 https://github.com/kkontras/MCR。
原文摘要 · Abstract (English)
Multimodal learning holds promise for richer information extraction by capturing dependencies across data sources. Yet, current training methods often underperform due to modality competition, a phenomenon where modalities contend for training resources leaving some underoptimized. This raises a pivotal question: how can we address training imbalances, ensure adequate optimization across all modalities, and achieve consistent performance improvements as we transition from unimodal to multimodal data? This paper proposes the Multimodal Competition Regularizer (MCR), inspired by a mutual information (MI) decomposition designed to prevent the adverse effects of competition in multimodal training. Our key contributions are: 1) A game-theoretic framework that adaptively balances modality contributions by encouraging each to maximize its informative role in the final prediction 2) Refining lower and upper bounds for each MI term to enhance the extraction of both task-relevant unique and shared information across modalities. 3) Proposing latent space permutations for conditional MI estimation, significantly improving computational efficiency. MCR outperforms all previously suggested training strategies and simple baseline, clearly demonstrating that training modalities jointly leads to important performance gains on both synthetic and large real-world datasets. We release our code and models at https://github.com/kkontras/MCR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。