arXiv:2505.03020cs.AI2025-05被引 4

多模态模型加数据能提性能,但缺数据时表现和公平性都下降。

The Multimodal Paradox: How Added and Missing Modalities Shape Bias and Performance in Multimodal AI

  • 在训练中加入新模态可稳定提升模型性能
  • 缺失模态导致推理时性能与公平性双双下降
  • 适合关注模型鲁棒性与公平性的医疗AI研究者

多模态学习融合图像、文本和结构化数据等多元信息,在高风险决策任务中表现优于单模态方法。然而,尽管性能提升是主要评估标准,偏见与鲁棒性问题常被忽视。本文探讨两个核心问题:(i) 添加新模态是否持续提升性能,并如何影响公平性,是缓解还是加剧偏见?(ii) 推理时模态缺失会对模型泛化能力(性能与公平性)造成何种影响?实验基于包含图像、时间序列和结构化信息的多模态医疗数据集,结果表明:训练中引入新模态能稳定提升性能,但公平性变化因评估指标与数据集而异;推理时缺失模态则显著降低性能与公平性,暴露实际部署中的脆弱性。

原文摘要 · Abstract (English)

Multimodal learning, which integrates diverse data sources such as images, text, and structured data, has proven superior to unimodal counterparts in high-stakes decision-making. However, while performance gains remain the gold standard for evaluating multimodal systems, concerns around bias and robustness are frequently overlooked. In this context, this paper explores two key research questions (RQs): (i) RQ1 examines whether adding a modality con-sistently enhances performance and investigates its role in shaping fairness measures, assessing whether it mitigates or amplifies bias in multimodal models; (ii) RQ2 investigates the impact of missing modalities at inference time, analyzing how multimodal models generalize in terms of both performance and fairness. Our analysis reveals that incorporating new modalities during training consistently enhances the performance of multimodal models, while fairness trends exhibit variability across different evaluation measures and datasets. Additionally, the absence of modalities at inference degrades performance and fairness, raising concerns about its robustness in real-world deployment. We conduct extensive experiments using multimodal healthcare datasets containing images, time series, and structured information to validate our findings.

多模态公平性鲁棒性医疗AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。