让大模型通过同行成功路径反推错误步骤，精准定位推理错误点。
Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
- 用同一训练组中成功的推理路径做教师，生成每步的指导信号。
- 在数学和代码任务上超越GRPO与同类自蒸馏方法，尤其在简答任务提升显著。
- 适合需要精细错误定位的长链条推理场景，如竞赛级数学求解。
基于可验证奖励的强化学习为每条推理轨迹分配单一标量，导致长推理链中词元级信用分配不明确。在线自蒸馏通过让同一模型作为教师并依赖特权信息来缓解此问题，生成密集的逐词元信号。然而，通常采用的正确答案仅提供终点提示：在简短回答任务中，教师在中间阶段缺乏路径引导能力。我们提出后见式自蒸馏（Hindsight Self-Distillation, HSD），将教师条件于当前训练组中一条成功同行推理路径。该路径是成功策略的精确样本，无需额外采样。相比仅提供最终答案，它提供了完整的成功延续，使信用信号集中于失败路径与成功同行之间的分歧位置。在Qwen3-8B与Qwen3-32B上，针对数学与代码基准测试，HSD优于GRPO变体及在线自蒸馏基线，在如AIME等简答任务上取得最大提升。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal. But the common choice of a ground-truth answer is only an endpoint cue: on terse-answer tasks, the teacher falls silent at the intermediate positions where path-level guidance matters most. We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout drawn from the current training group. Such a peer is an exact sample from the success-conditioned policy, requiring no additional sampled rollouts. By providing a full successful continuation rather than only the final answer, the resulting credit signal concentrates at the divergence position between a failed rollout and a successful peer. Across Qwen3-8B and Qwen3-32B on math and code benchmarks, HSD obtains the best result against GRPO variants and on-policy distillation baselines, with the largest gains on terse-answer tasks such as AIME.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。