arXiv:2607.28077cs.CL2026-07

让大模型推理更高效:智能筛选提示词,平衡重复使用与探索新思路。

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

  • 根据提示词历史表现动态分配生成预算,区分可复用与需探索的提示。
  • 在6个数学推理任务上提升2.6%~3.7%,训练速度更快。
  • 仅增加约2秒/步开销,适合资源有限的大模型强化学习训练。

强化学习结合可验证奖励(RLVR)能提升大语言模型的推理能力,但相同回报的提示组会浪费生成预算而无有效学习信号。预滚动提示选择可减少此类浪费,但现有方法难以平衡利用与探索:反复利用历史有效提示会缩小训练覆盖范围,而广泛探索又降低有效提示比例。为此,我们提出LEEPS,一种基于潜在空间的探索-利用提示采样器,自适应平衡已有有效提示的重用与对不确定提示的持续探索。LEEPS将候选提示分为利用和探索组合,并根据其近期非平凡比率动态分配滚动预算。它进一步利用表示空间邻居和历史回滚结果,优先选择可能产生非零奖励方差的不确定提示,使探索更精准且无需额外回滚。在六个数学推理基准上,LEEPS在两种模型规模下均取得最高平均分,相较最强基线分别提升2.6%和3.7%;在三个评估的分布外通用推理基准上也表现最优,且每训练步骤仅增加约2秒在线采样开销。代码已公开于https://github.com/ShuangLiangX/LEEPS。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6\% and 3.7\% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.

强化学习大模型推理提示优化高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。