arXiv:2605.16874cs.AI2026-05中稿 · ICML被引 2

通过修正少数关键决策词,能显著提升基础模型的推理能力。

Reasoning Can Be Restored by Correcting a Few Decision Tokens

论文配图:Reasoning Can Be Restored by Correcting a Few Decision Tokens
图 1 · 摘自论文原文
  • 识别出推理错误集中于早期规划类决策词,仅占生成词的8%
  • 在少量干预下,性能可接近甚至超过同规模推理模型
  • 适合希望低成本提升现有模型推理能力的研究者

大型推理模型(LRMs)在复杂推理基准上显著优于基础大语言模型(LLM),但其失败原因尚不明确。我们通过似然差异量化基础模型与强推理模型在词级分布上的分歧。结果显示,推理优势高度稀疏,集中在少数早期规划类决策词上:在 Qwen3-0.6B 上,约 8% 的生成词贡献了主要分歧,这些词集中于响应早期,规划类决策占比高达 17 倍,且对应基础模型高不确定性——表明基础模型主要在引导后续推理路径的早期规划点上失效。基于此,我们提出分歧引导的词干预方法:仅在高分歧位置由推理模型接管一个词,随后立即切换回基础模型。仅需少量干预预算,该方法即可大幅恢复性能,甚至超越同规模推理模型。代码已公开于 https://github.com/AlphaLab-USTC/RRTokenIntervention。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) substantially outperform their base LLM counterparts on challenging reasoning benchmarks, yet it remains poorly understood where base models go wrong during token-by-token generation and how to narrow this gap efficiently. We study the base-reasoning gap through quantifying token-level distributional disagreement between a base model and a stronger reasoning model using likelihood-based divergences. Across benchmarks, we find that the reasoning advantage is highly sparse and concentrates on a small set of early, planning-related decision tokens. For instance, on Qwen3-0.6B, only ~8% of generated tokens account for the salient disagreement, and these tokens concentrate early in the response, are strongly enriched in planning-related decisions (17x), and coincide with high base-model uncertainty -- suggesting that base models fail mainly at early planning points that steer the subsequent reasoning trajectory. Building on these findings, we propose disagreement-guided token intervention, a simple inference-time delegation scheme that performs a one-token takeover by the reasoning model only at high-disagreement positions and immediately switches back to the base model. With a small intervention budget, this sparse delegation substantially recovers and can even surpass the performance of a same-size reasoning model on challenging reasoning tasks. Code is available at https://github.com/AlphaLab-USTC/RRTokenIntervention.

推理增强模型干预小样本优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。