arXiv:2604.13371cs.CL2026-04被引 1

发现大模型在复杂逻辑任务中会突然失效,且越复杂越容易出错。

Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints

论文配图:Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints
图 1 · 摘自论文原文
  • 用9个可调复杂度的逻辑题测试大模型推理能力
  • 超过特定复杂度后准确率暴跌超50%,出现错误输出
  • 适合关注模型真实推理能力的研究者与评估者

大型语言模型(LLMs)常被视作具备强大推理能力,这由其在数学、逻辑和规划基准上的高表现所支持。然而,现有评估多依赖固定数据集的总体准确率,掩盖了推理行为随任务复杂度变化的动态过程。本文提出一个受控基准框架,系统评估大型推理模型(LRMs)在逐步增加复杂度下的鲁棒性。构建了九类经典推理任务:布尔可满足性、密码算术、图着色、河渡问题、汉诺塔、水罐问题、跳棋、数独和魔方,每项任务均可精确调节复杂度并保持语义不变。通过确定性验证器,对多个开源与专有LRMs在低、中、高复杂度区间进行评估,仅接受完全有效的解。结果揭示一致的相变式行为:低复杂度下准确率高,但超过任务特异性阈值后急剧下降。我们将其称为‘推理坍塌’。各任务中观察到显著准确率下降,常超50%,伴随推理轨迹不一致、约束违反、状态追踪丢失及自信错误输出。推理长度增加并不保证正确性提升,且某类问题的性能提升无法泛化至其他问题。研究强调需发展超越静态基准的评估方法,明确测量推理在可控复杂度下的鲁棒性。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly described as possessing strong reasoning capabilities, supported by high performance on mathematical, logical, and planning benchmarks. However, most existing evaluations rely on aggregate accuracy over fixed datasets, obscuring how reasoning behavior evolves as task complexity increases. In this work, we introduce a controlled benchmarking framework to systematically evaluate the robustness of reasoning in Large Reasoning Models (LRMs) under progressively increasing problem complexity. We construct a suite of nine classical reasoning tasks: Boolean Satisfiability, Cryptarithmetic, Graph Coloring, River Crossing, Tower of Hanoi, Water Jug, Checker Jumping, Sudoku, and Rubik's Cube, each parameterized to precisely control complexity while preserving underlying semantics. Using deterministic validators, we evaluate multiple open and proprietary LRMs across low, intermediate, and high complexity regimes, ensuring that only fully valid solutions are accepted. Our results reveal a consistent phase transition like behavior: models achieve high accuracy at low complexity but degrade sharply beyond task specific complexity thresholds. We formalize this phenomenon as reasoning collapse. Across tasks, we observe substantial accuracy declines, often exceeding 50%, accompanied by inconsistent reasoning traces, constraint violations, loss of state tracking, and confidently incorrect outputs. Increased reasoning length does not reliably improve correctness, and gains in one problem family do not generalize to others. These findings highlight the need for evaluation methodologies that move beyond static benchmarks and explicitly measure reasoning robustness under controlled complexity.

大模型推理复杂度测试推理崩溃

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。