让大模型自动生成提示，提升稀疏奖励下的强化学习效果
Self-Hinting Language Models Enhance Reinforcement Learning
- 模型自生成简洁提示（如计划）来增强策略多样性
- 在6个基准上平均提升2.0分，显著优于传统方法
- 无需额外信息即可部署，适合需要稳定训练的场景
组相对策略优化（GRPO）是目前对齐大语言模型与可验证目标的有效方法。但在稀疏终端奖励下，同一组内的多次采样常得相同奖励，导致相对优势消失，更新停滞。为此，我们提出自提示对齐的GRPO框架SAGE，通过在训练中注入特权提示（如计划或分解）重塑采样分布。对于每个输入x，模型先采样一个紧凑提示h，再基于(x,h)生成解τ。任务奖励R(x,τ)保持不变，仅提示提升组内结果多样性，防止奖励稀疏时优势坍塌。测试时设h为空，直接使用无提示策略。自适应采样多样提示能更好追踪学习瓶颈，优于固定提示。在3个大模型、6个基准上的实验表明，SAGE平均提升2.0（Llama-3.2-3B-Instruct）、1.2（Qwen2.5-7B-Instruct）和1.3（Qwen3-4B-Instruct）。代码已开源。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has recently emerged as a practical recipe for aligning large language models with verifiable objectives. However, under sparse terminal rewards, GRPO often stalls because rollouts within a group frequently receive identical rewards, causing relative advantages to collapse and updates to vanish. We propose self-hint aligned GRPO with privileged supervision (SAGE), an on-policy reinforcement learning framework that injects privileged hints during training to reshape the rollout distribution under the same terminal verifier reward. For each prompt $x$, the model samples a compact hint $h$ (e.g., a plan or decomposition) and then generates a solution $τ$ conditioned on $(x,h)$. Crucially, the task reward $R(x,τ)$ is unchanged; hints only increase within-group outcome diversity under finite sampling, preventing GRPO advantages from collapsing under sparse rewards. At test time, we set $h=\varnothing$ and deploy the no-hint policy without any privileged information. Moreover, sampling diverse self-hints serves as an adaptive curriculum that tracks the learner's bottlenecks more effectively than fixed hints from an initial policy or a stronger external model. Experiments over 6 benchmarks with 3 LLMs show that SAGE consistently outperforms GRPO, on average +2.0 on Llama-3.2-3B-Instruct, +1.2 on Qwen2.5-7B-Instruct and +1.3 on Qwen3-4B-Instruct. The code is available at https://github.com/BaohaoLiao/SAGE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。