测试多模态大模型看图写代码能力，发现闭源模型远超开源。
Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities
- 用流程图生成对应代码，评估模型逻辑理解与编程能力
- 闭源模型在难题上达79.3%正确率，开源模型仅15%
- 专为检测多模态推理瓶颈设计，适合研究模型泛化能力者
本文提出Code-Vision，一个用于评估多模态大语言模型（MLLMs）逻辑理解与代码生成能力的基准。该基准要求模型根据给定的流程图生成满足特定功能需求的正确程序，流程图直观表示目标算法或过程。Code-Vision包含三个子集：HumanEval-V、Algorithm和MATH，分别覆盖基础编程、算法与数学问题求解领域。我们对12个MLLMs进行了评估。实验结果表明，闭源与开源模型间性能差距显著：在难题上，GPT-4o达到79.3% pass@1，而表现最好的开源模型仅为15%。进一步分析显示，Code-Vision相比MMCode与MathVista等基准能带来独特挑战。我们还探究了开源模型表现不佳的原因。所有数据与代码已公开于https://github.com/wanghanbinpanda/CodeVision。
原文摘要 · Abstract (English)
This paper introduces Code-Vision, a benchmark designed to evaluate the logical understanding and code generation capabilities of Multimodal Large Language Models (MLLMs). It challenges MLLMs to generate a correct program that fulfills specific functionality requirements based on a given flowchart, which visually represents the desired algorithm or process. Code-Vision comprises three subsets: HumanEval-V, Algorithm, and MATH, which evaluate MLLMs' coding abilities across basic programming, algorithmic, and mathematical problem-solving domains. Our experiments evaluate 12 MLLMs on Code-Vision. Experimental results demonstrate that there is a large performance difference between proprietary and open-source models. On Hard problems, GPT-4o can achieve 79.3% pass@1, but the best open-source model only achieves 15%. Further experiments reveal that Code-Vision can pose unique challenges compared to other multimodal reasoning benchmarks MMCode and MathVista. We also explore the reason for the poor performance of the open-source models. All data and codes are available at https://github.com/wanghanbinpanda/CodeVision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。