用小模型预测提示难易度，高效提升大模型推理训练效果
Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models
- 用轻量生成模型基于优化历史做贝叶斯推断，预测提示难度
- 批量选择中等难度且多样提示，训练效率提升40%以上
- 模型可泛化到新提示，适合资源有限的强化学习训练场景
强化学习能提升大语言模型的推理能力，但因需大量采样而计算成本高。在线提示选择可通过优先选取信息量高的提示来提高训练效率。然而，现有方法要么依赖昂贵的精确评估，要么构建了无法跨提示泛化的专用预测模型。本文提出通用提示预测选择（GPS），利用轻量生成模型对共享优化历史进行贝叶斯推断，以估计提示难度。通过引入中等难度优先和历史锚定多样性，设计批次获取策略，实现高效提示选择。小型预测模型在测试时具备泛化能力，支持高效的计算资源分配。在多个推理基准上的实验表明，相比先进基线方法，GPS在训练效率、最终性能和测试时效率上均有显著提升。
原文摘要 · Abstract (English)
Reinforcement learning enhances the reasoning capabilities of large language models but often involves high computational costs due to rollout-intensive optimization. Online prompt selection presents a plausible solution by prioritizing informative prompts to improve training efficiency. However, current methods either depend on costly, exact evaluations or construct prompt-specific predictive models lacking generalization across prompts. This study introduces Generalizable Predictive Prompt Selection (GPS), which performs Bayesian inference towards prompt difficulty using a lightweight generative model trained on the shared optimization history. Intermediate-difficulty prioritization and history-anchored diversity are incorporated into the batch acquisition principle to select informative prompt batches. The small predictive model also generalizes at test-time for efficient computational allocation. Experiments across varied reasoning benchmarks indicate GPS's substantial improvements in training efficiency, final performance, and test-time efficiency over superior baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。