实验证明,思维链效果来自内容而非长度,关键在推理质量。
Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters

- 用同一模型生成长短不一的自然推理链,对比准确性
- 相同内容下,冗长文本仅带来1-4%的微小提升
- 适合关注推理质量而非字数的AI研究者
思维链提示能提升大模型推理能力,但其原因存在争议:是中间步骤携带了有用语义内容,还是单纯通过更多词元计算获得优势?我们从分布内证据和受控干预两方面展开分析。首先,在分布内实验中,对25个独立训练的模型重复提问,并配对同一模型生成的短与长推理链(遵循相同推理计划,无重写),结果表明额外词元对准确率基本无影响;盲评显示,任何微弱增益均与验证或检查内容相关,而非冗余度。其次,通过双验证器设计,在四个目标、八个基准上,使用数值掩码完成并采用分层自举置信区间,验证具有相同语义内容(事实、操作、中间值)但表达更冗长的链是否表现更好。结果显示,冗长链虽有轻微提升(32个测试单元中有25个为正向),但幅度有限(通常1-4点),且依赖于语言表达质量而非长度。在最大数值掩码下,效果放大至平均3.24倍,而长度匹配的非推理填充无法恢复该收益。综合表明,真正起作用的是额外词元所承载的推理与验证内容,而非数量本身——这一现象无法被纯计算量或纯语义内容解释。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditioning on more tokens buys extra computation before the model commits to an answer? We bring two lines of evidence to bear. First, in distribution: we repeatedly sample each model on the same question and pair a shorter with a longer of its own natural generations that follow the same reasoning plan, so nothing is rewritten and both traces are genuinely in-distribution. Across 25 models the extra tokens leave accuracy essentially unchanged for every independently-trained reasoner, and a blind analysis of the surplus tokens shows that what gain exists elsewhere tracks validation- and checking-content, not verbosity per se. Second, as a controlled intervention, we ask whether two traces expressing the same semantic content (the same facts, operations, and intermediate values, verified through directed acyclic graph equivalence) produce different outcomes when one is more verbose, using a dual-validator design across four targets and eight benchmarks with number-redacted completion and stratified bootstrap confidence intervals. Verbose traces do improve accuracy (25 of 32 benchmark-target cells are positive under at least one validator), but the effects are modest (typically 1-4 points) and depend on the quality of the verbose prose, not merely its length. Under maximum numerical redaction the effect is amplified (median 3.24x across four arithmetic benchmarks), and length-matched non-reasoning filler recovers none of it. Both lines converge: what matters is what the extra tokens do (the reasoning and validation content they carry), not how many there are, a picture neither a pure forward-pass-compute nor a pure semantic-content account fully explains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。