让模型通过画图来思考,测试视觉推理能力。
When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought
- 设计需要生成中间图像的推理任务,模拟人类'画图思考'过程。
- 提供546个带中间图像的多模态问题,验证模型表现。
- 视觉思维提示使模型平均提升33.7%,凸显图像对推理的关键作用。
我们提出MIRA,一个评估模型在生成中间视觉图像对推理至关重要的场景下的基准。与仅依赖文本的传统链式思维(CoT)不同,MIRA中的任务要求模型生成并利用草图、结构图或路径图等中间图像来引导推理过程,这更贴近人类通过'画图思考'解决复杂问题的方式。该基准聚焦于涉及复杂结构、空间关系或难以仅用语言表达的推理步骤的任务。为保证数据质量,MIRA包含546个多模态问题,附带中间视觉图像和最终答案。我们还提出统一的三层次评估协议:仅输入图像和问题;纯文本链式思维输入;以及带有标注图像线索和文本思维提示的视觉链式思维输入。通过报告不同k值下的pass@k和多数投票准确率,我们探查了模型在该基准上的上限。实验表明,现有多模态大模型(包括最强私有模型和强开源模型)在仅依赖文本提示时表现不佳。但当提供中间视觉线索时,模型性能显著提升,所有模型和任务的平均相对增益达33.7%。即使扩展搜索空间并设计与视觉链式思维对齐的文本提示,改进也有限。这些结果突显了想象中的视觉信息在成功推理中的核心作用。
原文摘要 · Abstract (English)
We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning. Unlike traditional CoT methods that rely solely on text, tasks in MIRA require models to generate and utilize intermediate images - such as sketches, structural diagrams, or path drawings - to guide their reasoning process. This setup closely mirrors how humans solve complex problems through "drawing to think". To solve this, MIRA focuses on tasks that are intrinsically challenging and involve complex structures, spatial relationships, or reasoning steps that are difficult to express through language alone. To ensure that our evaluation data is of high-quality, we include 546 multimodal problems, annotated with intermediate visual images and final answers. We also propose a unified evaluation protocol for MIRA that spans three levels of evaluation input: direct input with image and question only, text-only CoT input with image and thinking prompts, and Visual-CoT input with both annotated image clues and textual thinking prompts. To probe the upper bound of model capacity on our benchmark, we also report pass@k and majority voting accuracies under different k settings. Experimental results show that existing multimodal large language models, including strongest private models as well as strong open-weight models, perform poorly when relying solely on textual prompts. However, when intermediate visual cues are provided, model performance improves consistently, yielding an average relative gain of 33.7% across all models and tasks. We also probe the upper bound by expanding the search space and designing textual prompts aligned with Visual-CoT, but both yield only limited improvements compared to our Visual-CoT setting. These results underscore the critical role of imagined visual information in enabling successful reasoning on MIRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。