用代码提升大模型对科学图像的感知能力,解决其视觉推理短板。
CodePercept: Code-Grounded Visual STEM Perception for MLLMs
- 以可执行代码作为视觉感知媒介,精准捕捉科学图像语义。
- 构建100万条图文代码三元组数据集,提升模型对结构化图像理解。
- 提出新评测基准,通过代码还原评估视觉感知而非仅看解题正确率。
当多模态大模型在科学、技术、工程和数学(STEM)视觉推理中失败时,根本原因究竟是感知缺陷还是推理不足?通过独立调节感知与推理组件的系统性扩展分析,我们发现:提升感知能力始终优于提升推理能力,揭示感知是当前STEM视觉推理的主要瓶颈。基于此洞察,本文聚焦于通过代码增强多模态大模型的感知能力——可执行代码提供精确语义,天然契合STEM视觉的结构性特征。我们构建了ICC-1M,一个包含100万条图像-标题-代码三元组的大规模数据集,通过两种互补方法实现代码即感知范式:(1) 代码引导的标题生成,将可执行代码作为图像标题的真值,消除现有知识蒸馏方法中的幻觉;(2) 科学图像到代码的翻译任务,促使模型生成重建代码,缓解自然语言在感知增强中的歧义性。为验证该范式,我们进一步提出STEM2Code-Eval,一种新型基准,直接评估STEM领域内的视觉感知能力。不同于以往依赖解题准确率作为代理指标的方法,本基准要求通过生成可执行代码实现图像重建,提供确定性且可验证的评估方式。代码已开源:https://github.com/TongkunGuan/Qwen-CodePercept。
原文摘要 · Abstract (English)
When MLLMs fail at Science, Technology, Engineering, and Mathematics (STEM) visual reasoning, a fundamental question arises: is it due to perceptual deficiencies or reasoning limitations? Through systematic scaling analysis that independently scales perception and reasoning components, we uncover a critical insight: scaling perception consistently outperforms scaling reasoning. This reveals perception as the true lever limiting current STEM visual reasoning. Motivated by this insight, our work focuses on systematically enhancing the perception capabilities of MLLMs by establishing code as a powerful perceptual medium--executable code provides precise semantics that naturally align with the structured nature of STEM visuals. Specifically, we construct ICC-1M, a large-scale dataset comprising 1M Image-Caption-Code triplets that materializes this code-as-perception paradigm through two complementary approaches: (1) Code-Grounded Caption Generation treats executable code as ground truth for image captions, eliminating the hallucinations inherent in existing knowledge distillation methods; (2) STEM Image-to-Code Translation prompts models to generate reconstruction code, mitigating the ambiguity of natural language for perception enhancement. To validate this paradigm, we further introduce STEM2Code-Eval, a novel benchmark that directly evaluates visual perception in STEM domains. Unlike existing work relying on problem-solving accuracy as a proxy that only measures problem-relevant understanding, our benchmark requires comprehensive visual comprehension through executable code generation for image reconstruction, providing deterministic and verifiable assessment. Code is available at https://github.com/TongkunGuan/Qwen-CodePercept.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。