arXiv:2604.06799cs.CLcs.CY2026-04

用九维框架诊断大模型代数推理失败原因,发现工作内存是核心瓶颈。

Beyond Accuracy: Diagnosing Algebraic Reasoning Failures in LLMs Across Nine Complexity Dimensions

  • 构建九维独立变量的代数问题生成与验证系统,无需人工标注。
  • 所有模型在20-30个并行分支后崩溃,显示工作内存为不可逾越的硬约束。
  • 只需5个维度即可完整刻画模型代数推理能力,适合评估与对比。

代数推理仍是检验大语言模型的重要压力测试,但现有基准无法定位失败的具体原因。当模型解题失败时,单一准确率无法区分是表达式嵌套过深、操作符罕见、中间结果过多,还是依赖链过长所致。以往研究仅孤立分析单一失败模式,缺乏在严格控制条件下独立调节各复杂度因素的系统性框架。我们提出一个九维代数复杂度框架,每个维度独立变化而其余保持不变,并通过参数化流水线实现问题自动生成与验证,无需人工标注。每维均基于已知的模型失败模式,涵盖表达式嵌套深度、并行中间结果数量、子表达式复杂度、操作符难度及依赖推理链长度等结构性差异。我们在7个指令微调模型(8B至235B参数)上全面评估,发现工作内存是跨规模的主导瓶颈:所有模型在20至30个并行分支后性能骤降,指向架构层面的硬限制而非可解决的容量问题。进一步分析识别出五个最小且诊断充分的维度组合,可完整覆盖已知代数失败模式,提供模型代数推理能力的完整复杂度画像。

原文摘要 · Abstract (English)

Algebraic reasoning remains one of the most informative stress tests for large language models, yet current benchmarks provide no mechanism for attributing failure to a specific cause. When a model fails an algebraic problem, a single accuracy score cannot reveal whether the expression was too deeply nested, the operator too uncommon, the intermediate state count too high, or the dependency chain too long. Prior work has studied individual failure modes in isolation, but no framework has varied each complexity factor independently under strict experimental control. No prior system has offered automatic generation and verification of problems of increasing complexity to track model progress over time. We introduce a nine-dimension algebraic complexity framework in which each factor is varied independently while all others are held fixed, with problem generation and verification handled by a parametric pipeline requiring no human annotation. Each dimension is grounded in a documented LLM failure mode and captures a structurally distinct aspect of algebraic difficulty, including expression nesting depth, simultaneous intermediate result count, sub-expression complexity, operator hardness, and dependent reasoning chain length. We evaluated seven instruction-tuned models spanning 8B to 235B parameters across all nine dimensions and find that working memory is the dominant scale-invariant bottleneck. Every model collapses between 20 and 30 parallel branches regardless of parameter count, pointing to a hard architectural constraint rather than a solvable capacity limitation. Our analysis further identifies a minimal yet diagnostically sufficient subset of five dimensions that together span the full space of documented algebraic failure modes, providing a complete complexity profile of a model's algebraic reasoning capacity.

代数推理大模型评测工作内存复杂度分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。