arXiv:2604.00790cs.AI2026-04

让大模型通过自我纠错提升编程解题能力,效果超越更大模型。

RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning

  • 设计怀疑型智能体,反复验证并修正代码,保持对自身输出的质疑。
  • 仅用标准验证数据训练,40亿参数模型击败320亿参数模型。
  • 适合想用小模型实现高精度编程解题的研究者与开发者。

尽管大语言模型在竞赛编程等复杂推理任务中表现优异,现有方法多局限于单次尝试,忽视了其迭代优化的潜力。本文提出RefineRL,一种激发大模型自我修正能力的新方法。该方法引入两个核心创新:(1) 怀疑型智能体,配备本地执行工具,可针对竞赛编程题的公开测试用例验证生成的解法。该智能体始终对自身输出持怀疑态度,即使验证通过也强制进行严格自我修正。(2) 仅使用标准强化学习验证反馈数据(即问题与其可验证答案配对),实现对大模型的自我修正激励。在Qwen3-4B和Qwen3-4B-2507上的大量实验表明,经强化学习训练后,这些小型40亿参数模型结合怀疑型智能体不仅超越了更大的320亿参数模型,还接近2350亿参数模型的单次尝试性能。结果表明,自我修正在提升大模型推理能力方面具有巨大潜力,未来仍有显著发展空间。

原文摘要 · Abstract (English)

While large language models (LLMs) have demonstrated strong performance on complex reasoning tasks such as competitive programming (CP), existing methods predominantly focus on single-attempt settings, overlooking their capacity for iterative refinement. In this paper, we present RefineRL, a novel approach designed to unleash the self-refinement capabilities of LLMs for CP problem solving. RefineRL introduces two key innovations: (1) Skeptical-Agent, an iterative self-refinement agent equipped with local execution tools to validate generated solutions against public test cases of CP problems. This agent always maintains a skeptical attitude towards its own outputs and thereby enforces rigorous self-refinement even when validation suggests correctness. (2) A reinforcement learning (RL) solution to incentivize LLMs to self-refine with only standard RLVR data (i.e., problems paired with their verifiable answers). Extensive experiments on Qwen3-4B and Qwen3-4B-2507 demonstrate that our method yields substantial gains: after our RL training, these compact 4B models integrated with the Skeptical-Agent not only outperform much larger 32B models but also approach the single-attempt performance of 235B models. These findings suggest that self-refinement holds considerable promise for scaling LLM reasoning, with significant potential for further advancement.

竞赛编程自修正强化学习小模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。