提出真实场景下多模态情感分析的评测基准，揭示模型在缺失数据不均时的隐藏缺陷。
MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities
- 构建四种数据集的共享与非均衡缺失率测试协议
- 发现模型在非均衡缺失下存在显著模态不公平与优化失衡
- 提供可复现的工具链，适合研究多模态鲁棒性与公平性的学者
多模态情感计算支撑情感分析与情绪识别等关键任务。现有评估通常假设文本、语音和视觉模态均等可用，但实际应用中某些模态更易缺失或成本更高，导致非均衡缺失率及训练偏差，仅靠任务级指标无法揭示。本文提出 MissBench，一个标准化的多模态情感任务基准与框架，在四个常用情感与情绪数据集上统一定义共享与非均衡缺失率协议。同时引入两项诊断指标：模态公平指数（MEI）衡量不同模态在各类缺失配置下的贡献公平性；模态学习指数（MLI）通过训练中模态相关模块的梯度范数对比，量化优化不平衡程度。在代表性方法族上的实验表明，看似在均匀缺失下鲁棒的模型，在非均衡条件下仍表现出显著的模态不公与优化失衡。这些发现使 MissBench 及其两个指标成为在现实不完整模态场景下压力测试与分析多模态情感模型的实用工具。代码已公开于：https://anonymous.4open.science/r/MissBench-4098/
原文摘要 · Abstract (English)
Multimodal affective computing underpins key tasks such as sentiment analysis and emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a benchmark and framework for multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality settings.For reproducibility, we release our code at: https://anonymous.4open.science/r/MissBench-4098/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。