构建首个覆盖全科目的本科物理推理评测基准，推动大模型在物理思维上的突破。
UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
- 设计涵盖13个学科的5520道中英文物理题，覆盖七种答案类型与四类推理能力。
- 31个主流大模型最高仅达49.8%准确率，远低于数学推理表现，凸显物理建模短板。
- 首创基于规则的自动化判分系统（MARJ），有效避免数据泄露，提升评估可靠性。
大型语言模型（LLMs）在解决复杂推理任务方面展现出惊人能力，尤其是在数学领域。然而，物理推理因其独特性而受到的关注较少，现有评测基准在覆盖本科物理的广度与深度上存在明显不足。为此，我们提出UGPhysics——一个专为评估大模型在本科物理推理能力而设计的大规模、综合性基准。该基准包含5,520道本科水平的物理题目，涵盖13个学科，提供中英文双语版本，包含七种不同答案类型和四种不同的物理推理技能，并经过严格筛选以防止数据泄露。此外，我们开发了面向物理问题的答案正确性评估专用的模型辅助规则判断（MARJ）流水线，确保评估准确性。对31个主流大模型的评估显示，最高整体准确率为49.8%（由OpenAI-o1-mini实现），表明当前模型在物理推理能力上仍显著落后于数学推理，亟需更强的物理理解与建模能力。我们希望UGPhysics与MARJ能推动未来人工智能在物理推理领域的进步。代码与数据已公开于 https://github.com/YangLabHKUST/UGPhysics。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable capabilities in solving complex reasoning tasks, particularly in mathematics. However, the domain of physics reasoning presents unique challenges that have received significantly less attention. Existing benchmarks often fall short in evaluating LLMs' abilities on the breadth and depth of undergraduate-level physics, underscoring the need for a comprehensive evaluation. To fill this gap, we introduce UGPhysics, a large-scale and comprehensive benchmark specifically designed to evaluate UnderGraduate-level Physics (UGPhysics) reasoning with LLMs. UGPhysics includes 5,520 undergraduate-level physics problems in both English and Chinese, covering 13 subjects with seven different answer types and four distinct physics reasoning skills, all rigorously screened for data leakage. Additionally, we develop a Model-Assistant Rule-based Judgment (MARJ) pipeline specifically tailored for assessing answer correctness of physics problems, ensuring accurate evaluation. Our evaluation of 31 leading LLMs shows that the highest overall accuracy, 49.8% (achieved by OpenAI-o1-mini), emphasizes the necessity for models with stronger physics reasoning skills, beyond math abilities. We hope UGPhysics, along with MARJ, will drive future advancements in AI for physics reasoning. Codes and data are available at https://github.com/YangLabHKUST/UGPhysics .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。