构建细粒度几何题数据集，评估大模型画辅助线与长步骤推理能力
GeoLaux: A Benchmark for Evaluating MLLMs' Geometry Performance on Long-Step Problems Requiring Auxiliary Lines
- 设计包含2186道题的细粒度数据集，覆盖平均6.51步、最长24步的推理过程
- 41.8%题目需画辅助线，模型在此类问题上性能普遍下降超50%
- 提示中间步骤可提升正确率，直接给答案反而削弱推理过程理解
几何问题求解（GPS）对多模态大语言模型（MLLMs）在图示理解、知识应用、长步骤推理和辅助线构造方面构成重大挑战。然而，现有基准测试缺乏对需辅助线的长步骤问题的细粒度评估。为此，我们提出GeoLaux，一个包含2186道计算与证明题的细粒度标注数据集，平均解题步骤为6.51步，最长达24步，其中41.8%的问题需要构造辅助线。基于该数据集，我们对23个主流MLLM进行了五维度综合评估。结果显示：第一，模型在长步骤问题上的表现显著低于短步骤问题，18个模型性能下降超过50%；第二，增强模型对辅助线的理解、意识与构造能力，对整体几何推理至关重要；第三，有限的答案提示能有效提升推理过程正确性，而直接给出答案则导致模型忽略中间步骤。GeoLaux不仅可用于评测MLLM的几何推理能力，也为模型优化提供方向。数据与代码已开源于https://github.com/Candice-yu/GeoLaux。
原文摘要 · Abstract (English)
Geometry problem solving (GPS) poses significant challenges for Multimodal Large Language Models (MLLMs) in diagram comprehension, knowledge application, long-step reasoning, and auxiliary line construction. However, current benchmarks lack fine-grained evaluation for long-step problems necessitating auxiliary construction. To address these limitations, we present GeoLaux, a fine-grained annotated dataset comprising 2186 calculation and proof problems. It features long-step reasoning (with an average solution length of 6.51 steps, maximum of 24 steps) and auxiliary line construction (required in 41.8% of problems). Building on the dataset, we conduct a comprehensive five-dimensional evaluation of 23 leading MLLMs. The evaluation yields three pivotal findings: First, models perform significantly worse on long-step problems compared to short-step ones, with 18 models exhibiting a performance drop of over 50%. Second, it is crucial to enhance models' understanding, awareness, and proficiency in auxiliary line construction, which is vital for overall geometric reasoning. Third, limited answer hints effectively improve process correctness, whereas explicit answers lead models to neglect intermediate reasoning steps. These findings position GeoLaux both to benchmark MLLMs geometry reasoning abilities and to guide their improvement. Data and code are available at https://github.com/Candice-yu/GeoLaux
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。