让零方差提示也参与训练，提升大模型推理能力。
No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
- 从无差异反馈的提示中提取学习信号，直接奖励正确、惩罚错误。
- 在六项数学推理任务上，准确率最高提升8.61点，通过率提升7.77点。
- 适合做强化学习优化大模型推理的研究者与开发者。
基于可验证奖励的强化学习（RLVR）是提升大语言模型（LLM）推理能力的强大框架。然而，现有方法如GRPO仅利用响应结果存在正确性差异的问题，忽略所有响应获得相同奖励的“零方差提示”。本文认为这类提示并非无用，反而能提供有意义的反馈。为此，提出一种新算法RL-ZVP，从零方差提示中提取学习信号。该方法直接奖励正确性并惩罚错误，无需对比响应，同时结合词元级特征调节反馈，保留丰富细微信息。在六个数学推理基准上，相比GRPO，RL-ZVP准确率最高提升8.61点，通过率提升7.77点，且始终优于过滤零方差提示的其他基线。结果表明，在RLVR中挖掘零方差提示的潜力具有重要意义。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful framework for improving the reasoning abilities of Large Language Models (LLMs). However, current methods such as GRPO rely only on problems where the model responses to the same input differ in correctness, while ignoring those where all responses receive the same reward -- so-called zero-variance prompts. In this work, we argue that such prompts are not useless but can, in fact, provide meaningful feedback for policy optimization. To this end, we introduce Reinforcement Learning with Zero-Variance Prompts (RL-ZVP), a novel algorithm that extract learning signals from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes errors even without contrasting responses, modulating feedback with token-level characteristics to preserve informative, nuanced signals. Across six math reasoning benchmarks, RL-ZVP achieves significant improvements of up to 8.61 points in accuracy and 7.77 points in pass rate over GRPO, while consistently outperforming other baselines that filter out zero-variance prompts. These results highlight the untapped potential of learning from zero-variance prompts in RLVR. The project page is available at https://bltnynk.github.io/publications/rl-zvp/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。