arXiv:2412.11088cs.AIcs.CV2024-12被引 8

大模型仅凭图像就能解图与树结构题,挑战传统考试形式。

Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Models

  • 用9072个图像样本测试大模型解图/树结构题能力
  • GPT-4o在树结构上达87.6%准确率,Gemini 1.5 Flash在图上56.2%
  • 为教育评估和编程教学提供新思路

近年来生成式AI的发展引发了教育界对学术诚信的担忧。除了擅长解决编程题和文本选择题外,已有研究发现大型多模态模型(LMMs)仅通过代码块图像即可解答Parsons问题。然而这类问题仍本质上依赖文本转换能力。本文进一步探究LMMs仅基于图像解决图与树数据结构问题的能力。为此,我们构建并评估了一个包含9,072个样本的新型基准数据集,涵盖多种图与树任务,用于测试GPT-4o、GPT-4v、Gemini 1.5 Pro、Gemini 1.5 Flash、Gemini 1.0 Pro Vision及Claude 3模型家族的表现。结果表明,GPT-4o在树结构任务上表现最佳,准确率达87.6%;Gemini 1.5 Flash在图结构任务上表现最优,准确率为56.2%。研究揭示了结构与视觉差异对模型性能的影响。本工作不仅引入一个可复现的LMM基准,还凸显了大模型在解决复杂计算问题上的潜力,对教学与评估具有重要启示。

原文摘要 · Abstract (English)

Recent advancements in generative AI systems have raised concerns about academic integrity among educators. Beyond excelling at solving programming problems and text-based multiple-choice questions, recent research has also found that large multimodal models (LMMs) can solve Parsons problems based only on an image. However, such problems are still inherently text-based and rely on the capabilities of the models to convert the images of code blocks to their corresponding text. In this paper, we further investigate the capabilities of LMMs to solve graph and tree data structure problems based only on images. To achieve this, we computationally construct and evaluate a novel benchmark dataset comprising 9,072 samples of diverse graph and tree data structure tasks to assess the performance of the GPT-4o, GPT-4v, Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 1.0 Pro Vision, and Claude 3 model families. GPT-4o and Gemini 1.5 Flash performed best on trees and graphs respectively. GPT-4o achieved 87.6% accuracy on tree samples, while Gemini 1.5 Flash, achieved 56.2% accuracy on graph samples. Our findings highlight the influence of structural and visual variations on model performance. This research not only introduces an LMM benchmark to facilitate replication and further exploration but also underscores the potential of LMMs in solving complex computing problems, with important implications for pedagogy and assessment practices.

多模态模型图结构教育评估代码理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。