arXiv:2605.18851cs.LG2026-05被引 1

用可学习的语言反馈替代传统奖励,提升大模型推理能力。

STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning

论文配图:STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
图 1 · 摘自论文原文
  • 通过联合训练生成器与生成式验证器,用语言反馈指导中间步骤修正。
  • 在多个推理任务上超越现有方法,零通过率问题也能有效学习。
  • 适合需要持续优化推理过程的AI研究者与工程师。

近期强化学习进展表明其在激发大语言模型(LLM)推理能力方面的潜力。然而,现有逐步方法依赖昂贵的人工标注,限制了领域覆盖;而标量奖励则造成信息瓶颈,难以改善中间决策。基于语言批评的替代方法虽提供更丰富的文本反馈,但依赖冻结或外部批评者,难以实现持续策略改进。本文提出一种名为STRIDE的新训练框架,将过程监督从标量奖励转向可学习的逐步语言反馈。具体地,仅使用结果奖励联合训练生成器与生成式验证器,无需外部标注;验证器的逐步语言批评能明确指出并解释错误,引导生成器在中间步骤修正推理路径。轨迹重定向设计确保即使在噪声或次优反馈下也能安全提升策略。在多个推理基准上的实验表明,STRIDE显著优于当前最佳基线,并在零通过率问题上实现突破——这是标量方法因无学习信号而无法处理的情况,验证了可学习语言反馈对增强LLM推理的有效性。

原文摘要 · Abstract (English)

Recent advances in Reinforcement Learning (RL) have underscored its potential for incentivizing reasoning capabilities of Large Language Models (LLMs). However, existing step-level efforts suffer from costly annotations that limit domain coverage, while scalar scores further impose an information bottleneck, offering insufficient semantic bandwidth to improve intermediate decisions. Alternative language-critique approaches, which rely on frozen or external critics, provide richer textual feedback but lack the scalability needed for sustained policy improvement. In this work, we propose language-driven stepwise trajectory redirection, termed as STRIDE, a novel training framework that shifts process supervision from scalar rewards to learnable stepwise language feedback. Specifically, we co-train a generator and a generative verifier using only outcome-based rewards, eliminating external annotations, while delivering sustained policy improvement through jointly aligned verifier training. The verifier's stepwise language critiques explicitly localize and explain failures, enabling the generator to redirect reasoning trajectories at intermediate steps toward alternative decisions. The trajectory redirection design guarantees harmless policy improvement, even under noisy or suboptimal verifier feedback. Experiments on diverse reasoning benchmarks show that STRIDE significantly outperforms state-of-the-art baselines, as well as achieving breakthroughs on zero-pass-rate problems where scalar methods yield no learning signal in our ablation studies, demonstrating the effectiveness of learnable stepwise language feedback for enhancing LLM reasoning.

大模型推理强化学习语言反馈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。