评测大模型在复杂图表编程任务中的视觉推理能力，发现顶尖模型仅36.8%正确率。
HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks
- 设计六类编码任务，结合图表与代码签名测试模型理解力
- 22个大模型实测，最优表现仅36.8%通过率，空间与动态关系是主要短板
- 适合关注多模态推理、编程理解的AI研究者与开发者
理解与推理图表是人类智能的核心能力。尽管大型多模态模型（LMMs）在各类任务中表现优异，现有基准仍缺乏对图表解读与推理能力的全面评估，尤其在编程场景中。我们提出HumanEval-V，一个由人工标注的编码任务基准，涵盖六类任务，全面评估多样化的视觉推理能力。每项任务均配有精心设计的图表，搭配函数签名与测试用例，采用创新的代码生成任务深入考察模型对图表的理解。通过对22个LMM的广泛实验，我们发现即使性能最佳的模型也仅达到36.8% pass@1，表明当前模型在空间变换、拓扑关系和动态模式等人类本能掌握的能力上仍有显著不足。该结果为提升LMM视觉推理能力提供了重要启示。代码与基准已开源：https://github.com/HumanEval-V/HumanEval-V-Benchmark。
原文摘要 · Abstract (English)
Understanding and reasoning over diagrams is a fundamental aspect of human intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive capabilities across various tasks, existing benchmarks lack comprehensive evaluation of their diagram interpretation and reasoning abilities, particularly in coding contexts. We present HumanEval-V, a rigorous benchmark of human-annotated coding tasks that spans six task types and evaluates diverse visual reasoning capabilities. Each task features carefully crafted diagrams paired with function signatures and test cases, employing novel code generation tasks to thoroughly assess models' diagram comprehension. Through extensive experiments with 22 LMMs, we find that even top-performing models achieve modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1, highlighting substantial room for improvement. Our analysis reveals that current LMMs struggle with spatial transformations, topological relationships, and dynamic patterns that humans find intuitive. These findings provide valuable insights for advancing LMMs' visual reasoning abilities. We have open-sourced our code and benchmark at https://github.com/HumanEval-V/HumanEval-V-Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。