让大模型生成能改进策略的自然语言反馈，提升强化学习效率。
RL with Learnable Textual Feedback: A Bilevel Approach
- 设计双向优化框架，让批评者生成能提升策略的反馈
- 2B模型在MATH-500上达46.6%，优于3B基线
- 适合追求高效推理训练的大模型研究者
基于可验证奖励的强化学习能提升大语言模型的推理能力，但终端奖励稀疏时学习效率仍低。为此，研究者提出使用文本反馈的强化学习方法，由评论模型生成自然语言反馈来指导推理模型（即执行者），以补充标量奖励。然而现有方法通常将反馈视为固定或辅助信息，忽略了关键特性：反馈不仅要正确，更应在上下文中真正提升策略表现。这促使我们提出可学习文本反馈的强化学习范式。由于反馈的可学习性与有效性依赖于策略的学习能力，该问题本质上是双层优化。本文将其形式化为斯塔克尔伯格双层规划，并提出双层自然语言演员-评论者（Bi-NAC）算法，联合训练能生成改善奖励的反馈的评论者与能利用反馈的执行者。在MATH-500、MBPP和GPQA数据集上，Bi-NAC在样本和参数效率上均优于强化学习及固定评论者基线：我们的2B模型在MATH-500上达到46.6%，超越3B GRPO基线（41.4%）；6B模型在GPQA上达49.3%，超过7B GRPO基线（43.6%）。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards can improve LLM reasoning, but learning remains sample-inefficient when terminal rewards are sparse. This has motivated a growing line of work on RL with textual feedback, where a critic model generates natural language feedback to guide a reasoning model (the actor), augmenting scalar rewards with richer learning signals. However, existing methods typically treat feedback as fixed or auxiliary, which misses a key property: feedback should not merely be correct, but should improve the policy (actor model) when provided in context. This motivates a paradigm of learnable textual feedback for RL. Yet the learnability and usefulness of feedback depend on the policy's ability to learn from it, making RL with learnable feedback an inherently bilevel problem. We formalize this coupling as a Stackelberg bilevel program and derive Bilevel Natural Language Actor-Critic (Bi-NAC), which jointly trains a critic to generate reward-improving feedback and an actor to exploit it. Across MATH-500, MBPP, and GPQA, Bi-NAC improves sample and parameter efficiency over RL and fixed-critic baselines: our 2B model outperforms the 3B GRPO baseline, achieving 46.6% versus 41.4% on MATH-500, while our 6B model surpasses the 7B GRPO baseline, achieving 49.3% versus 43.6% on GPQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。