链式思维能救命也能添乱，关键看任务是否超了模型单次处理的容量。
When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning
- 用串行深度梯度分析发现，模型单次处理能力不足时，链式思维可缓解瓶颈。
- 在高串行深度任务中，链式思维提升准确率54到68个百分点，效果显著。
- 低复杂度任务中链式思维无效但无害，适合评估模型真实推理能力者参考。
链式思维（CoT）提示是否普遍提升大模型推理能力尚存争议。本文基于H_dp带宽界限理论框架（Chen et al., 2024），揭示尽管该界仅在极限长上下文成立，却识别出真实架构瓶颈：超过Transformer单次处理能力的串行计算必须外化，而链式思维正是实现这一外化的机制。核心发现为基准内串行深度梯度：单次处理（无CoT）准确率随每项任务串行深度单调下降，而链式思维则近似深度不变。我们在三个指令微调模型（Qwen-2.5-7B/32B、Llama-3.1-8B）和五个标准NLP基准上测量了链式思维效果。在高串行深度的P-complete任务（GSM8K、MATH）中，链式思维带来+54至+68个百分点的准确率恢复；在浅层TC^0任务（MMLU、ARC）中，链式思维结构冗余（Δ在[0.0, +4.6]百分点，无显著负效应）——尽管无链式思维基线高达95%（如ARC），可能反映数据污染，故此零效应并非纯净的架构测试。中间类任务L（HumanEval）呈现模型规模依赖性转变：32B模型+23.2，8B模型+9.1，7B模型-28.7。跨基准深度-恢复相关性为Spearman rho = 0.661（p = 0.007, n = 15），15个基准级McNemar检验中有9个经伯恩费尔德校正后显著。研究预注册于OSF，结果表明链式思维非通用增强工具，而是带宽绕行机制：仅对超出单次处理容量的任务有效，对已适配的任务则冗余。
原文摘要 · Abstract (English)
It is widely assumed that chain-of-thought (CoT) prompting universally improves LLM reasoning. We investigate this through the conceptual framework of the H_dp bandwidth bound (Chen et al., 2024): although the formal bound binds only asymptotically (at astronomically large prompt lengths), it identifies a real architectural bottleneck -- serial computation exceeding a transformer's single-pass capacity must be externalised, which is what CoT does. Our central finding is a within-benchmark serial-depth gradient: single-pass (no-CoT) accuracy degrades monotonically with per-item serial depth, while CoT is approximately depth-invariant. We measure CoT effects across three instruction-tuned models (Qwen-2.5-7B/32B, Llama-3.1-8B) and five standard NLP benchmarks at practical context lengths. On high-depth P-complete tasks (GSM8K, MATH), CoT gives a +54 to +68 pp recovery gap across all models. On shallow TC^0 tasks (MMLU, ARC), CoT is structurally redundant (Delta in [0.0, +4.6] pp, no significant negative effect) -- though high no-CoT baselines (up to 95% on ARC) may reflect contamination, so this null is not a clean architectural test. The intermediate class L (HumanEval) shows a model-size-dependent transition: +23.2 pp (32B), +9.1 pp (8B), -28.7 pp (7B). The cross-benchmark depth-recovery correlation is Spearman rho = 0.661 (p = 0.007, n = 15); 9 of 15 benchmark-level McNemar tests are significant after Bonferroni correction. Pre-registered on OSF, our results indicate that CoT is not a universal reasoning enhancer but acts as a bandwidth bypass: it helps serial computation that strains single-pass capacity and is redundant for tasks that already fit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。