评测大模型主动发现并修正用户输入错误的能力,揭示其推理与回答不一致的隐藏缺陷。
MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

- 构建细粒度错误分类体系,涵盖跨模态矛盾等12类输入缺陷
- 14个主流多模态模型在细微视觉异常上纠错率不足50%
- 发现普遍存在的'一致性缺口':模型会自省错误却仍选择合规输出
随着多模态大语言模型(MLLMs)演变为复杂交互助手,其可靠性不仅取决于执行指令,还依赖于对指令的验证能力。我们定义主动批判(Proactive Critique)为模型在无额外提示下自主识别、分析并修复错误输入的能力。现有评估多局限于理想场景或简单拒绝行为,忽视了主动纠错机制。为此,我们提出MMPCBench,一个全面评估MLLM主动批判能力的框架。该框架包含4类主要错误类型、12个子类别,涵盖跨模态矛盾至缺失视觉前提等问题。采用分层评估协议衡量模型的错误检测、诊断与修复表现,并引入对齐感知指标评估内部推理与最终输出的一致性。在14个主流MLLM上的测试显示,模型在处理细微视觉异常时存在明显弱点。特别地,我们发现普遍存在‘一致性缺口’:模型在内部推理中常能正确识别和分析错误,却在最终输出中压制这些有效洞察以优先保证响应合规性。代码与数据已开源。
原文摘要 · Abstract (English)
As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating them. We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts. However, evaluations mainly test models under ideal circumstances or simple refusal behaviors, largely ignoring active error processing. To fill this gap, we propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence. It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises. We adopt a hierarchical evaluation protocol to measure models' error detection, diagnosis and resolution performance, and apply alignment-aware metrics to assess the coherence between internal reasoning and final responses. Tests on 14 mainstream MLLMs show obvious weaknesses in proactive critique, especially in dealing with subtle visual anomalies. Notably, we identify a pervasive "consistency gap": reasoning models can often correctly identify and analyze errors during internal reasoning yet suppress these valid insights in final outputs to prioritize response compliance. The code and data is available at https://github.com/ALIENS32/MMPCBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。