提出新指标MCS，评估多模态数据融合质量。
Good Scores, Bad Data: A Metric for Multimodal Coherence
- 设计四维分解的MCS指标，自动学习权重评估融合一致性。
- 在1000张图像上验证，MCS比任务准确率更敏感（Spearman rho=0.093）。
- 无需人工标注，可定位具体出问题的模态维度，适合模型调试。
多模态AI系统通常通过下游任务准确率评估，但高准确率不代表输入数据一致。模型在视觉问答（VQA）中表现良好，其多模态输入仍可能自相矛盾。我们提出多模态一致性评分（MCS），独立于下游模型评估融合质量。MCS将一致性分解为身份、空间、语义和决策四个维度，权重通过Nelder-Mead优化学习。在1000张Visual Genome图像上使用DETR、CLIP和ViLT进行评估，并在150张COCO图像上无重训练验证。三种融合架构下，MCS对质量差异的判别力高于仅依赖任务准确率（斯皮尔曼相关系数0.093 vs. 0.071）。扰动实验表明各维度对特定故障模式响应独立，无交叉干扰。MCS轻量高效，无需人工标注，不仅能发现错误，还能指出具体出错维度。
原文摘要 · Abstract (English)
Multimodal AI systems are evaluated by downstream task accuracy, but high accuracy does not mean the underlying data is coherent. A model can score well on Visual Question Answering (VQA) while its inputs contradict each other. We introduce the Multimodal Coherence Score (MCS), a metric that evaluates fusion quality independent of any downstream model. MCS decomposes coherence into four dimensions, identity, spatial, semantic, and decision, with weights learned via Nelder-Mead optimization. We evaluate on 1,000 Visual Genome images using DETR, CLIP, and ViLT, and validate on 150 COCO images with no retraining. Across three fusion architectures, MCS discriminates quality with higher sensitivity than task accuracy alone (Spearman rho = 0.093 vs. 0.071). Perturbation experiments confirm each dimension responds independently to its failure mode with zero cross-talk. MCS is lightweight, requires no human annotation, and tells you not just that something broke, but what broke.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。