用少量优质示范样例,让大模型逐步提升解题能力
From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
- 通过专家解答片段引导模型生成,动态优化训练样本
- 在数学推理任务上收敛更快,性能上限更高
- 适合希望高效提升模型推理能力的研究者
强化学习结合可验证奖励(RLVR)近期提升了大语言模型的推理能力。现有工作多关注算法设计、数据筛选与奖励构造,本文从样本中心视角出发,提出LPPO(学习进展与前缀引导优化)框架,聚焦如何有效利用少量高质量示范样例,而非单纯增加数据量。首先,受人类解题提示启发,提出前缀引导采样,通过引入专家示范中的部分解题前缀,动态指导策略生成,尤其适用于复杂问题。其次,借鉴人类根据自身能力聚焦关键问题的机制,设计学习进展加权策略,基于样本级通过率的指数移动平均估计学习进展,增强有进步潜力样本的影响,弱化停滞样本权重。在数学推理基准测试中,该方法显著优于强基线,实现更快收敛和更高性能天花板。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has recently advanced the reasoning capabilities of large language models (LLMs). While prior work has emphasized algorithmic design, data curation, and reward shaping, we investigate RLVR from a sample-centric perspective and introduce LPPO (Learning-Progress and Prefix-guided Optimization), a framework of progressive optimization techniques. Our work addresses a critical question: how to best leverage a small set of trusted, high-quality demonstrations, rather than simply scaling up data volume. First, motivated by how hints aid human problem-solving, we propose prefix-guided sampling, an online data augmentation method that incorporates partial solution prefixes from expert demonstrations to guide the policy, particularly for challenging instances. Second, inspired by how humans focus on important questions aligned with their current capabilities, we introduce learning-progress weighting, a dynamic strategy that adjusts each training sample's influence based on model progression. We estimate sample-level learning progress via an exponential moving average of per-sample pass rates, promoting samples that foster learning and de-emphasizing stagnant ones. Experiments on mathematical-reasoning benchmarks demonstrate that our methods outperform strong baselines, yielding faster convergence and a higher performance ceiling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。