arXiv:2606.24902cs.DLcs.AI2026-06

揭示大模型数学推理失败的四大模式,发现错误常藏于无依据的默认假设。

Failure Modes of Large Language Models on Research-Level Mathematics: A Taxonomy and an Empirical Characterisation

  • 基于十道难题的诊断,提炼出四类错误模式:编造引用、偷换前提等。
  • 8个生成证明中无一编造引用,但全都有未加证明的关键假设。
  • 提出新检测工具可精准识别隐蔽的‘前提偷渡’,适合研究可信AI的学者。

《首次证明》基准测试将十个科研级数学问题交给当前最强的公开大模型,结果它们并非沉默,而是自信且流畅地犯错。本文探究其原因:基于该基准附录A的逐题分析,作者归纳出四种失败模式——引用伪造(F1)、前提偷渡(F2)、静默重述问题(F3)和局部到全局不兼容(F4)。随后,对Gemini 2.5 Flash在三个问题上生成的八份单次推理证明进行审计,使用专门设计的工具检测F1和F2。核心发现令人不安:八份证明中无人编造引用,但每一份均存在关键结论被当作‘基本结果’或‘标准论证’直接断言,毫无依据。这种现象即为F2——前提偷渡,而它本质上无法通过引用验证来发现。本文提出的前提审计工具在该语料库中达到100%精确率(5个裁判确认的警报均为真阳性),证明级别召回率为50%。该分类与审计共同表明,长期目标应是构建推理时能预防此类错误的系统,而非事后检测。

原文摘要 · Abstract (English)

The "First Proof" benchmark [1] posed ten research-level mathematics questions to the strongest publicly available LLMs and found them consistently wrong-not silent, but confidently, fluently wrong. This paper asks why. Working from the per-question post-mortems in First Proof's Appendix A, I identify four failure modes: citation fabrication (F1), premise smuggling (F2), silent problem reformulation (F3), and local-to-global compatibility gaps (F4). I then audit eight one-shot proofs generated by Gemini 2.5 Flash on Questions 1, 2, and 5 of the benchmark, using two instruments built specifically to surface F1 and F2. The central finding is uncomfortable for anyone who sees retrieval-augmented generation (RAG) as the obvious fix: not one of the eight proofs contained a confirmed fabricated citation, yet every single one contained at least one load-bearing claim asserted as a "fundamental result" or "standard argument" with no justification attached. That failure mode-F2, premise smuggling-is invisible to citation verification by design. A premise-audit instrument I introduce flags it at 100% precision (5/5 judge-confirmed flags are true positives) and 50% proof-level recall in this corpus. The taxonomy and the audit together suggest that the right long-term objective is building inference-time pipelines that prevent these failure modes from occurring, not just detecting them after the fact. Index Terms--Large language models, mathematical reasoning, hallucination, premise smuggling, failure-mode taxonomy.

大模型数学推理幻觉检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。