训练大模型时,只需用推理路径的起点和终点,就能达到与完整路径相同的效果。
Revisiting Complete Reasoning Traces for Post-Training
- 只用推理路径的起点和终点进行训练,效果优于完整路径。
- 中间步骤贡献极低,删除80%中间token后准确率仅下降2.3%。
- 适合想压缩训练数据、提升效率的研究者或工程师。
大语言模型常通过预收集的推理轨迹进行后训练以提升推理能力。这些轨迹因路径复杂而冗长,常包含偏离主路径的枝节。然而,现有研究未充分探讨模型是否真正受益于完整轨迹的后训练(如监督微调)。我们的初步研究发现,完整轨迹带来的增益有限,而部分轨迹在严重截断下仍有效。通过注意力分析和受控删词实验,我们发现中间token对最终推理质量影响甚微。这表明,模型可利用内部知识推断缺失步骤,从而从端点生成连贯推理。此外,基于端点训练能稳定改变推理行为,并对强化学习或在线蒸馏等方法同样有益,提示需重新审视完整推理轨迹的价值。代码已开源:https://github.com/naver-ai/revisiting-trace。
原文摘要 · Abstract (English)
Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability. Such trajectories tend to be long due to complex, interwoven paths, which often include detours on the path toward the answer. However, it has been underexplored whether LLMs indeed benefit from learning complete trajectories in post-training, such as supervised fine-tuning (SFT). Starting from our pilot study, we find that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation. We analyze redundancy in reasoning trajectories through attention-based analyses and controlled token-removal studies, both of which show that intermediate tokens contribute minimally to final reasoning quality. This suggests that avoiding redundant information may allow LLMs to internally infer coherent alternatives by inferring missing steps from their internal knowledge, given known trajectory endpoints. Furthermore, we show that training LLMs using endpoints leads to consistent changes in reasoning behavior, and that it also benefits post-training methods based on reinforcement learning or on-policy distillation, highlighting the need to revisit complete reasoning traces. Code is available at https://github.com/naver-ai/revisiting-trace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。