揭露大模型在4×4以上矩阵计算中从算错到胡编的崩溃规律
LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning

- 构建3×3至5×5矩阵的9类660题基准,精细追踪失败模式
- 发现4×4是分水岭:小矩阵错在算术错误,大矩阵转为直接胡编
- 揭示模型因记忆局限而非知识不足而失效,适合评估数学推理能力
我们提出LinAlg-Bench,一个诊断性基准,评估10个前沿大语言模型在3×3、4×4和5×5矩阵上的结构化线性代数计算。涵盖9类任务和660个SymPy验证问题,共评测6,600次模型输出。除二分类准确率外,引入三阶段自动化取证流程,将1,156次失败分类为10类主要错误标签及细粒度子类型。结果表明,大模型数学错误并非随机,而是受算法类型与矩阵维度结构性约束。核心发现:4×4为行为临界点——低于此规模,模型因符号追踪失误、算术漂移、奇偶错误等执行错误失败;高于此规模,失败转向计算放弃,模型通过工具角色扮演、约束一致幻觉和结构化幻觉编造答案。该从虚构到放弃的转变在所有模型层级与架构中普遍出现,暗示工作内存限制而非知识缺失。此外,三种在3×3不存在但在4×4和5×5出现的误差类型呈尺度涌现特征。研究还发现解题策略僵化可近乎完美预测5×5行列式准确率,并首次记录约束感知型幻觉这一新型结构化幻觉失败模式。所有数据、模型输出、错误标签及评审流程已公开。
原文摘要 · Abstract (English)
We introduce LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier large language models on structured linear algebra computation across a strict dimensional gradient of 3x3, 4x4, and 5x5 matrices. Spanning 9 task types and 660 SymPy-certified problems, the benchmark exhaustively evaluates 6,600 model outputs. Beyond binary accuracy, LinAlg-Bench introduces a three-stage automated forensic pipeline classifying 1,156 failures into ten primary error tags with fine-grained subtypes, revealing that LLM mathematical failure is not random but structurally constrained by algorithm type and matrix dimension. Our central finding is a sharp behavioral threshold at 4x4 scale: below it, models fail through execution errors -- sign tracking failures, arithmetic drift, and parity errors; above it, failure transitions to computational abandonment, with models fabricating responses through tool roleplay, constraint-consistent confabulation, and structured hallucination rather than attempting computation. This fabrication-to-abandonment transition is near-universal across all model tiers and architectures, suggesting a working memory limit rather than a knowledge gap, supported by three scale-emergent error types absent at 3x3 but present at 4x4 and 5x5. We further show that solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy, document constraint-aware confabulation as a novel structured hallucination failure mode, and release all data, model outputs, error labels, and judge pipeline publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。