让大模型学会自我批评，自动优化代码生成结果。
Teaching Language Models to Critique via Reinforcement Learning
- 用强化学习训练模型生成能提升代码正确率的反馈。
- 在挑战性数据集上实现最高106.1%的性能提升。
- 无需人工标注，适合希望自动优化生成质量的开发者。
让大语言模型（LLMs）学会对自身输出进行批判与改进，对构建可迭代优化的系统至关重要，但受限于准确判断和可行建议的能力。本文研究代码生成任务中的LLM批评者，提出一种名为$ exttt{CTRL}$的框架——通过强化学习训练批评模型，在不依赖人类标注的前提下，生成能最大化固定生成器纠错表现的反馈。实验表明，采用$ exttt{CTRL}$训练的批评模型显著提升了通过率，并缓解了错误累积问题，适用于基础与更强的生成器模型。此外，这些批评模型可作为准确的生成式奖励模型，支持测试时通过迭代批判-修正实现规模扩展，在复杂代码生成基准上实现最高达106.1%的相对性能提升。
原文摘要 · Abstract (English)
Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, we study LLM critics for code generation and propose $\texttt{CTRL}$, a framework for $\texttt{C}$ritic $\texttt{T}$raining via $\texttt{R}$einforcement $\texttt{L}$earning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with $\texttt{CTRL}$ significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1% relative improvements across challenging code generation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。