让大模型在答错时自我修正,提升复杂问题求解能力
LatentRevise: Learning from Zero-Hit Reasoning

- 通过修正错误推理路径的潜在表示,恢复训练信号
- 修正后推理路径更长且能自省,正确率显著提升
- 适合想改进大模型逻辑推理能力的研究者
强化学习结合可验证奖励(RLVR)受限于难以生成正确轨迹的困难提示,导致采样在合理预算内无法覆盖正确路径,使策略更新缺乏有效信号。我们将这类零命中提示视为RLVR的采样边界,在此区域新推理行为最宝贵却最不可能被采样。重要的是,失败的推演过程具有信息价值:揭示了模型推理中的错误位置。我们提出首个一阶潜在修正方法LatentRevise,针对失败推演与正确答案作为锚点,优化推理前缀的输入嵌入,沿两个互补梯度方向调整:远离失败延续方向,靠近正确答案。优化过程被约束于模型词表嵌入的凸包内,确保每次更新都指向真实词嵌入而非任意特征方向。实验发现,经修正的前缀生成的延续路径更长、具备自反思能力,并能达成原推演遗漏的正确答案。将这些轨迹用于训练,显著提升SFT与RLVR在数学基准测试上的表现。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by hard prompts on which correct trajectories have low probability, so sampling misses them within a practical budget and leaves the policy update with little useful signal. We frame such zero-hit prompts as RLVR's sampling frontier, where new reasoning behavior is most valuable yet least likely to be sampled. Importantly, failed rollouts can be informative: they expose where the model's reasoning went wrong. We introduce LatentRevise, a first-order latent revision method that recovers training signal for this zero-hit regime. Given a failed rollout and the gold answer as an anchor, LatentRevise optimizes the input embeddings of its reasoning prefix under two complementary gradients, moving the prefix away from the failed continuation and toward the gold answer. The optimization is constrained to the convex hull of the model's vocabulary embeddings, so each update moves the latent toward a real token embedding rather than an arbitrary feature direction. We find that continuations from the revised prefix lengthen, exhibit self-reflection, and reach correct answers missed by the original rollouts. Used as training data, these trajectories improve SFT and RLVR on math benchmarks over standard baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。