提出理论解释为何压缩推理步骤会失效，并设计新方法实现高效推理。
Chain Of Thought Compression: A Theoretical Analysis
- 从高阶逻辑依赖角度分析推理压缩的理论难点
- 在新基准上验证压缩导致性能下降，速度提升54.4倍且保持原精度
- 适合研究高效大模型推理与理论机制的学者
链式思维（CoT）虽能激发大语言模型的高级推理能力，但因生成额外标记而带来高昂计算成本。近期研究发现，将推理步骤压缩为隐状态（即隐式CoT压缩）可节省标记，但其内在机制尚不明确。本文首次对内化中间推理步骤的学习难度进行理论分析，引入‘阶数-r交互’概念，证明高阶逻辑依赖的学习信号会指数级衰减，跳过中间步骤将导致不可逾越的高阶交互障碍。为实证验证，我们构建了NatBool-DAG这一挑战性基准，强制执行不可简化的逻辑推理并消除语义捷径。基于理论发现，提出ALiCoT（对齐隐式链式思维）框架，通过使隐状态分布与中间推理状态对齐，克服信号衰减问题。实验表明，ALiCoT实现了54.4倍的速度提升，同时性能与显式CoT相当。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) has unlocked advanced reasoning abilities of Large Language Models (LLMs) with intermediate steps, yet incurs prohibitive computational costs due to generation of extra tokens. Recent studies empirically show that compressing reasoning steps into latent states, or implicit CoT compression, offers a token-efficient alternative. However, the mechanism behind CoT compression remains unclear. In this paper, we provide the first theoretical analysis of the difficulty of learning to internalize intermediate reasoning steps. By introducing Order-r Interaction, we prove that the learning signal for high-order logical dependencies exponentially decays to solve irreducible problem, where skipping intermediate steps inevitably leads to high-order interaction barriers. To empirically validate this, we introduce NatBool-DAG, a challenging benchmark designed to enforce irreducible logical reasoning and eliminate semantic shortcuts. Guided by our theoretical findings, we propose ALiCoT (Aligned Implicit CoT), a novel framework that overcomes the signal decay by aligning latent token distributions with intermediate reasoning states. Experimental results demonstrate that ALiCoT successfully unlocks efficient reasoning: it achieves a 54.4x speedup while maintaining performance comparable to explicit CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。