通过联合优化策略与提示，解决大模型推理中难以样本的奖励消失问题。
P^2O: Joint Policy and Prompt Optimization
- 交替更新连续策略与离散提示，用进化算法寻找有效推理路径。
- 在难样本上恢复学习信号，性能提升最高达9.5%，超越双倍采样预算的基线。
- 适合追求鲁棒推理与自动对齐的大模型研究者，尤其关注稀疏奖励场景。
基于可验证奖励的强化学习（RLVR）能增强大语言模型（LLM）的推理能力，但在“难样本”上所有回放均失败时，优势值塌陷导致关键学习信号消失。单纯增加回放预算无法有效缓解此问题。本文提出联合策略与提示优化（P²O），通过连续策略更新与离散提示演化的交替机制，利用GEPA算法为难样本发现成功推理提示。借助上下文蒸馏，模型将提示带来的收益直接内化到参数中，无需推理时显式提示。实验表明，P²O恢复了关键优势信号，显著优于标准GRPO，在双倍回放预算下仍超越基线，最终实现强泛化能力，性能最高提升9.5%。研究揭示了稀疏奖励环境下传统探索方法的局限性，展示了进化算法与强化学习融合的潜力，建立了离散语义搜索与连续参数更新协同的自增强范式，推动自主化大模型对齐。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) enhances Large Language Model (LLM) reasoning but suffers from advantage collapse on ``hard samples'' where all rollouts fail. This lack of variance eliminates crucial learning signals. For these intractable samples, simply scaling up rollout budgets offers limited gains. We introduce Joint Policy and Prompt Optimization (P$^2$O) to mitigate this collapse by alternating continuous policy updates with discrete prompt evolution. P$^2$O leverages the GEPA algorithm to discover successful reasoning prompts for intractable instances. Via context distillation, the model internalizes these prompt-induced gains directly into its parameters, removing the need for inference-time prompting. Empirically, P$^2$O restores critical advantage signals, significantly outperforming standard GRPO and surpassing baselines with doubled rollout budgets, ultimately yielding strong out-of-distribution generalization and an up to $9.5\%$ performance improvement. Our findings expose the limits of standard exploration in sparse-reward environments, illuminating the potential of unifying evolutionary algorithms with reinforcement learning. This integration of discrete semantic search and continuous parameter updates establishes a self-reinforcing paradigm for autonomous LLM alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。