arXiv:2606.18910cs.LGcs.CL2026-06

通过修正与验证增强训练,提升大模型推理的多步纠错能力。

REVES: REvision and VErification--Augmented Training for Test-Time Scaling

论文配图:REVES: REvision and VErification--Augmented Training for Test-Time Scaling
图 1 · 摘自论文原文
  • 分两阶段迭代优化:在线生成修正提示,集中训练错误识别与答案改写。
  • 在LiveCodeBench上比强化学习基线提升6.5分,比传统多轮训练高4.0分。
  • 适用于代码、数学及约束满足类问题,适合小模型高效部署。

通过序列修正实现的测试时扩展已成为增强大语言模型推理能力的有效范式。然而,传统训练方法主要优化单次输出目标,与多步推理动态存在根本性错配。尽管近期工作将其视为多轮强化学习,但现有方法直接优化长轨迹,未能充分利用中间步骤中高质量的‘近似正确’答案所蕴含的纠错信息。本文提出一种两阶段迭代框架,交替进行在线数据/提示增强与策略优化。将成功恢复轨迹中的中间步骤(‘近似正确’答案)转化为独立的修正与验证提示,使训练聚焦于有效答案转换与错误识别。该方法实现了高效的离线策略数据生成,并显著降低长周期采样的计算开销。在LiveCodeBench上,使用公开测试用例作为反馈,相较强化学习基线提升6.5分,比标准多轮训练高4.0分。在代码以外任务中,本方法在圆排列问题上达到此前报告的最优结果,仅使用4B规模基础模型,且远少于大型演化搜索系统所需采样次数。在真值验证下的数学任务中进一步证实了更强的纠错能力。该方法还可泛化至无分布外的约束满足谜题(如n_queens、mini_sudoku),其中正确性完全由问题约束定义。代码已开源:https://github.com/yxliu02/REVES.git。

原文摘要 · Abstract (English)

Test-time scaling via sequential revision has emerged as a powerful paradigm for enhancing Large Language Model (LLM) reasoning. However, standard post-training methods primarily optimize single-shot objectives, creating a fundamental misalignment with multi-step inference dynamics. While recent work treats this as multi-turn reinforcement learning (RL), conventional approaches optimize over the multi-step trajectories directly, failing to further exploit the high-quality mistakes in intermediate steps that model can learn from correcting them. We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization. By converting the intermediate steps (``near-miss'' answers) in the successful recovery trajectories into decoupled revision and verification prompts, our approach concentrates training on both effective answer transformation and error identification. This approach enables efficient off-policy data generation and reduces the computational overhead of long-horizon sampling compared to standard multi-turn RL. On LiveCodeBench, using publicly available test cases as feedback, we observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training. Beyond coding, our approach matches the previously reported SOTA result on circle packing while using the smallest base model (4B) and far fewer rollouts than the much larger evolutionary search systems. Math results under ground-truth verification further confirm improved correction ability. It also generalizes to out-of-distribution constraint-satisfaction puzzles such as n\_queens and mini\_sudoku, where correctness is defined entirely by problem constraints. Code is available at https://github.com/yxliu02/REVES.git.

大模型推理测试时扩展纠错机制强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。