构建细粒度跨模态推理评估基准,揭示大模型在理科任务中的真实推理能力
Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks

- 设计严格互补的图文输入,防止单模态作弊
- 仅38.29%准确率暴露模型依赖文本推理的短板
- 适合研究多模态推理机制与模型评估的学者
多模态大语言模型虽展现潜力,但专业领域评估仍具挑战。STEM推理因反馈可验证而成为理想测试场景,但现有基准常因模态冗余导致单模态捷径,且仅关注最终答案准确率,忽视推理过程。为此,我们提出StepSTEM:涵盖数学、物理、化学、生物和工程的283道研究生级题目,通过严格数据筛选确保文本与视觉输入严格互补。我们还设计了通用的步骤级评估框架,使用动态规划对齐预测推理步骤与多个参考解法。在多种模型上实验显示,当前多模态模型仍严重依赖文本推理,即使Gemini 3.1 Pro和Claude Opus 4.6也仅达38.29%准确率。结果凸显真实跨模态推理的巨大提升空间,并确立StepSTEM作为细粒度多模态推理评估基准。代码已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have shown promising reasoning abilities, yet evaluating their performance in specialized domains remains challenging. STEM reasoning is a particularly valuable testbed because it provides highly verifiable feedback, but existing benchmarks often permit unimodal shortcuts due to modality redundancy and focus mainly on final-answer accuracy, overlooking the reasoning process itself. To address this challenge, we introduce StepSTEM: a graduate-level benchmark of 283 problems across mathematics, physics, chemistry, biology, and engineering for fine-grained evaluation of cross-modal reasoning in MLLMs. StepSTEM is constructed through a rigorous curation pipeline that enforces strict complementarity between textual and visual inputs. We further propose a general step-level evaluation framework for both text-only chain-of-thought and interleaved image-text reasoning, using dynamic programming to align predicted reasoning steps with multiple reference solutions. Experiments across a wide range of models show that current MLLMs still rely heavily on textual reasoning, with even Gemini 3.1 Pro and Claude Opus 4.6 achieving only 38.29% accuracy. These results highlight substantial headroom for genuine cross-modal STEM reasoning and position StepSTEM as a benchmark for fine-grained evaluation of multimodal reasoning. Source code is available at https://github.com/lll-hhh/STEPSTEM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。