从因果视角缓解多模态对齐中的视觉偏见,提升模型鲁棒性。
Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
- 通过因果分析分离视觉模态的直接效应,仅依赖间接影响进行对齐
- 在低相似度、高噪声等困难场景下性能超越14种先进方法
- 适合处理图像质量差或数据稀缺的多模态知识图谱对齐任务
多模态实体对齐(MMEA)旨在从不同多模态知识图谱(MMKGs)中检索等价实体,是关键的信息检索任务。现有研究探索了多种融合范式和一致性约束以提升对齐效果,却忽视了视觉模态并不总能带来正向贡献。实证发现,图像相似度低的实体常导致性能下降,暴露出过度依赖视觉特征的局限。我们认为模型可能对视觉模态存在偏见,形成绕过语义的图像匹配捷径。为此,我们提出一种反事实去偏框架CDMEA,从因果视角解决视觉模态偏见问题。该方法通过估计双模态的总效应(TE),剔除视觉模态的自然直接效应(NDE),使模型仅基于总间接效应(TIE)进行预测,从而有效利用视觉与图结构模态,降低视觉偏见。在9个基准数据集上的大量实验表明,CDMEA优于14种最先进方法,尤其在低相似度、高噪声及低资源场景表现突出。
原文摘要 · Abstract (English)
Multi-Modal Entity Alignment (MMEA) aims to retrieve equivalent entities from different Multi-Modal Knowledge Graphs (MMKGs), a critical information retrieval task. Existing studies have explored various fusion paradigms and consistency constraints to improve the alignment of equivalent entities, while overlooking that the visual modality may not always contribute positively. Empirically, entities with low-similarity images usually generate unsatisfactory performance, highlighting the limitation of overly relying on visual features. We believe the model can be biased toward the visual modality, leading to a shortcut image-matching task. To address this, we propose a counterfactual debiasing framework for MMEA, termed CDMEA, which investigates visual modality bias from a causal perspective. Our approach aims to leverage both visual and graph modalities to enhance MMEA while suppressing the direct causal effect of the visual modality on model predictions. By estimating the Total Effect (TE) of both modalities and excluding the Natural Direct Effect (NDE) of the visual modality, we ensure that the model predicts based on the Total Indirect Effect (TIE), effectively utilizing both modalities and reducing visual modality bias. Extensive experiments on 9 benchmark datasets show that CDMEA outperforms 14 state-of-the-art methods, especially in low-similarity, high-noise, and low-resource data scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。