arXiv:2604.16421cs.CLcs.AI2026-04

测试大模型在几何题不同表达形式下的鲁棒性,发现表现差异高达14个百分点。

Measuring Representation Robustness in Large Language Models for Geometry

  • 设计新评测框架GeoRepEval,对比同一题的欧氏、坐标、向量三种表达
  • 向量形式准确率最低,即使控制复杂度后仍差至Invariance@3=0.044
  • 高阶模型通过提示改写可提升52%准确率,说明问题在表达敏感而非能力不足

大语言模型在数学推理评估中日益重要,但其对等价问题表示方式的鲁棒性仍不清楚。几何问题可表现为欧氏、坐标或向量形式,而现有基准仅固定格式报告准确率,隐含表示不变性假设,掩盖了仅由表示变化引发的失败。本文提出GeoRepEval,一种表示感知的评估框架,从问题层面测量正确性、不变性和一致性,结合严格答案匹配、自助置信区间、配对McNemar检验、表示翻转分析和表面复杂度回归控制。我们证明Invariance@3指标可分解准确率中的鲁棒与脆弱成分,且受最弱表示限制。在158道精选高中几何题(474个实例)上评估11个LLM,发现仅因表示选择导致的准确率差距达14个百分点。向量形式始终为薄弱环节,即使控制长度与符号复杂度后,Invariance@3仍低至0.044。转换-求解提示干预使高容量模型向量准确率最高提升52个百分点,表明失败源于表示敏感而非能力不足;但低容量模型无改善,暗示深层局限。结果表明当前模型依赖表示特定启发式,而非抽象几何推理。所有数据集、提示与脚本已开源于https://github.com/vedjaw/GeoRepEval。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains poorly understood. In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms, but existing benchmarks report accuracy on fixed formats, implicitly assuming representation invariance and masking failures caused by representational changes alone. We propose GeoRepEval, a representation-aware evaluation framework that measures correctness, invariance, and consistency at the problem level across parallel formulations, combining strict answer matching, bootstrap confidence intervals, paired McNemar tests, representation-flip analyses, and regression controls for surface complexity. We prove that our Invariance@3 metric decomposes accuracy into robust and fragile components and is bounded by the weakest representation. Evaluating eleven LLMs on 158 curated high-school geometry problems (474 instances), we find accuracy gaps of up to 14 percentage points induced solely by representation choice. Vector formulations emerge as a consistent failure point, with Invariance@3 as low as 0.044 even after controlling for length and symbolic complexity. A convert-then-solve prompting intervention improves vector accuracy by up to 52 percentage points for high-capacity models, suggesting that failures reflect representation sensitivity rather than inability; however, low-capacity models show no gains, indicating deeper limitations. These results suggest that current models rely on representation-specific heuristics rather than abstract geometric reasoning. All datasets, prompts, and scripts are released at https://github.com/vedjaw/GeoRepEval.

大模型评测几何推理表示鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。