arXiv:2501.06143physics.ed-phcs.AI2025-01被引 32

GPT-4o在多语言物理概念测试中表现优异,但图像理解能力较弱。

Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories

  • 将物理题以图片形式输入,测试AI多模态理解能力。
  • 在除实验技能外的所有领域均优于普通本科生。
  • 对图像信息的解读能力明显弱于纯文本题目。

我们研究了基于大语言模型的人工智能系统GPT-4o在多语言、多学科物理概念测验中的表现。测验题来自PhysPort网站,涵盖经典物理(力学、电磁学、光学、热学)、相对论、量子力学、天文学、数学及实验技能等主题。不同于以往仅使用文本的研究,我们将题目以图片形式上传,模拟学生实际看到的试卷,评估系统的多模态能力。结果显示,各学科表现存在差异,实验技能为最弱项;在语言方面,英语及欧洲语言表现最佳。值得注意的是,题目难度与语言无关。与现有学生表现研究对比发现,除实验技能外,GPT-4o在所有领域均超过平均水平的本科生。此外,依赖图像解析的题目表现明显低于纯文本题目。尽管本研究探索性地展示了GPT-4o在物理教育中的潜力,但也强调教师需培养学生批判性评估AI输出的能力,审慎调整课程,并关注人工智能融入带来的公平性问题。

原文摘要 · Abstract (English)

We investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover classical physics topics such as mechanics, electromagnetism, optics, and thermodynamics, as well as relativity, quantum mechanics, astronomy, mathematics, and laboratory skills. Unlike previous text-only studies, we uploaded the inventories as images to reflect what a student would see on paper, thereby assessing the system's multimodal functionality. Our results indicate variation in performance across subjects, with laboratory skills standing out as the weakest. We also observe differences across languages, with English and European languages showing the strongest performance. Notably, the relative difficulty of an inventory item is largely independent of the language of the survey. When comparing AI results to existing literature on student performance, we find that the AI system outperforms average post-instruction undergraduate students in all subject categories except laboratory skills. Furthermore, the AI performs worse on items requiring visual interpretation of images than on those that are purely text-based. While our exploratory findings show GPT-4o's potential usefulness in physics education, they highlight the critical need for instructors to foster students' ability to critically evaluate AI outputs, adapt curricula thoughtfully in response to AI advancements, and address equity concerns associated with AI integration.

多模态AI物理教育GPT-4oAI评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。