解决多模态数据缺失不均衡问题，提升模型鲁棒性
BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates
- 通过全局上下文重校准特征，统一不同缺失模式下的表示基础
- 从分布与空间双角度调节梯度，平衡各模态学习动态
- 无需改动主干结构，适配多种多模态任务，尤其情绪识别
多模态学习常因信息丰富模态主导优化而产生不平衡，当各模态缺失概率不同时（即不均衡缺失率，IMR），该问题更为严重，扭曲表征学习与梯度动态。本文从训练过程出发，提出BALM——一种模型无关的即插即用框架，在IMR下实现平衡的多模态学习。框架包含两个互补模块：特征校准模块（FCM）利用全局上下文重新校准单模态特征，建立跨异构缺失模式的共享表示基础；梯度重平衡模块（GRM）从分布与空间双重视角调节梯度大小与方向，平衡各模态学习动态。BALM可无缝集成于多种骨干网络，包括多模态情绪识别（MER）模型，且不改变其架构。在多个MER基准测试中，实验结果表明，BALM在多样缺失与不平衡设置下均显著提升模型鲁棒性与性能。代码已公开于https://github.com/np4s/BALM_CVPR2026.git。
原文摘要 · Abstract (English)
Learning from multiple modalities often suffers from imbalance, where information-rich modalities dominate optimization while weaker or partially missing modalities contribute less. This imbalance becomes severe in realistic settings with imbalanced missing rates (IMR), where each modality is absent with different probabilities, distorting representation learning and gradient dynamics. We revisit this issue from a training-process perspective and propose BALM, a model-agnostic plug-in framework to achieve balanced multimodal learning under IMR. The framework comprises two complementary modules: the Feature Calibration Module (FCM), which recalibrates unimodal features using global context to establish a shared representation basis across heterogeneous missing patterns; the Gradient Rebalancing Module (GRM), which balances learning dynamics across modalities by modulating gradient magnitudes and directions from both distributional and spatial perspectives. BALM can be seamlessly integrated into diverse backbones, including multimodal emotion recognition (MER) models, without altering their architectures. Experimental results across multiple MER benchmarks confirm that BALM consistently enhances robustness and improves performance under diverse missing and imbalance settings. Code available at: https://github.com/np4s/BALM_CVPR2026.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。