测试不同推理长度对大模型答案质量的影响,发现早停未必更准。
Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces

- 用限定词数或简洁指令控制推理长度,对比相同步长下的表现。
- 512词时准确率提升3.8个百分点,但2048词时效果不明确。
- 早期停止常因低投入导致答案概率集中,不一定更可靠。
大语言模型常需冗长推理,增加成本与延迟。通过在198个GPQA Diamond和500个MMLU-Pro问题上进行配对实验,测试了针对Qwen3-14B的数值/简洁提示,以及gpt-oss-20b和-120b的训练努力设置。Qwen提示使推理长度缩短12%-17%,匹配词数下准确率变化小且不一致。简洁指令在512词时使MMLU-Pro准确率提升3.8个百分点,其中2.7个百分点来自未完成运行。2048词时收益不确定。对于gpt-oss,低/中等投入的已完成推理答案比匹配步长的高投入答案准确率高出14.5-26.3个百分点。512词优势主要来自低投入提前完成;未完成运行间差异较小且不一致。错误早停答案常将概率集中在选中选项,表明提前终止并不统一提升概率质量。实验表明,严格时限可能偏好低投入或简洁指令,而允许高投入完成则可恢复更高最终准确率。评估应分别报告截止前正确完成情况、停止时的答案、未完成运行间的差异及正确答案的概率。
原文摘要 · Abstract (English)
Large language models often reason at length before answering, increasing cost and latency. Prompts and trained settings can shorten this reasoning, but a shorter trace may only show that the model stopped sooner. Here, we evaluate paired runs of the same question at matched reasoning horizons across 198 GPQA Diamond and 500 MMLU-Pro questions. We test a numeric/concision prompt that announces a token limit for Qwen3-14B and the trained effort settings of gpt-oss-20b and -120b. The Qwen prompt shortens reasoning traces by 12-17%, while accuracy changes at matched token limits are small and mixed. A concise/early-answer instruction raises MMLU-Pro accuracy by 3.8 percentage points at 512 tokens, including +2.7 points when both runs are unfinished. Its gain at 2,048 tokens is uncertain. For gpt-oss, candidate-logit answers from completed low- and medium-effort reasoning are 14.5-26.3 points more accurate than matched-horizon high-effort answers. Most of the 512-token advantage comes from lower effort finishing earlier, while differences among unfinished runs are smaller and mixed. Wrong early answers often concentrate probability on the chosen option, so earlier stopping does not uniformly improve probability quality. In these tests, a tight deadline can favor lower effort or a concise instruction, whereas allowing high effort to finish can recover higher final accuracy. Evaluations should report correct completion before a deadline, the answer obtained when a run is stopped, differences among unfinished runs, and probability assigned to the correct answer separately.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。