新基准测试发现大模型视觉理解严重失效，正确答案背后常有误读。
Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
- 设计多维度视觉评测基准，含2612个问题、1758张图像，覆盖2D/3D任务
- 顶尖闭源模型平均准确率不足50%，远低于人类96.49%的水平
- 揭示模型在细节感知和注意力分配上的根本缺陷，适合研究者改进视觉能力
多模态大语言模型（MLLMs）虽具推理潜力，但其视觉感知能力仍是关键瓶颈。初步研究表明，某一领先模型在回答推理问题时，29%的正确答案仍存在视觉感知错误。为此，我们提出「Do You See Me」，一个包含1758张图像和2612个问题的可扩展基准，涵盖七类受人类心理学启发的子任务，支持2D与3D场景，具备可控复杂度，用于严格评估MLLM视觉能力。对3个闭源及5个开源主流模型的测评显示，人类准确率达96.49%，而顶级模型平均低于50%。随着任务复杂度提升，差距迅速扩大（如视觉形态恒常性子任务中，从12%降至45%）。进一步分析表明，失败源于视觉注意力错位及细粒度特征内部表示不稳，尤其在编码器补丁分辨率处表现脆弱。这凸显了构建真正鲁棒视觉感知模型的紧迫性。数据集、代码与评估脚本已开源：https://github.com/microsoft/Do-You-See-Me。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) show reasoning promise, yet their visual perception is a critical bottleneck. Strikingly, MLLMs can produce correct answers even while misinterpreting crucial visual elements, masking these underlying failures. Our preliminary study on a joint perception-reasoning dataset revealed that for one leading MLLM, 29% of its correct answers to reasoning questions still exhibited visual perception errors. To systematically address this, we introduce "Do You See Me", a scalable benchmark with 1,758 images and 2,612 questions. It spans seven human-psychology inspired subtasks in 2D and 3D, featuring controllable complexity to rigorously evaluate MLLM visual skills. Our findings on 3 leading closed-source and 5 major open-source models reveal a stark deficit: humans achieve 96.49% accuracy, while top MLLMs average below 50%. This performance gap widens rapidly with increased task complexity (e.g., from 12% to 45% in the visual form constancy subtask). Further analysis into the root causes suggests that failures stem from challenges like misallocated visual attention and the instability of internal representations for fine-grained details, especially at or below encoder patch resolution. This underscores an urgent need for MLLMs with truly robust visual perception. The benchmark dataset, source code and evaluation scripts are available at https://github.com/microsoft/Do-You-See-Me.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。