用极少数据和计算量,让大模型推理更高效。
Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
- 动态选一个最有价值样本做策略更新,降低计算开销。
- 仅需少量训练样本即可达到优秀推理效果,样本效率高。
- 适合资源有限但需强推理能力的模型优化场景。
大型语言模型在复杂推理任务中表现卓越,基于可验证奖励的强化学习(RLVR)已成为对齐模型推理链的有效框架。然而,RLVR成本高昂,需大量奖励信号和高额推演开销。本文重新审视RLVR中的数据与计算效率问题,首先建立解锁推理能力所需的样本复杂度下界,并实证表明仅需极少数训练实例即可实现优异性能。为此,提出动态单次策略精炼(DoPR),一种基于不确定性感知的强化学习策略:每批次动态选择一个信息量高的样本进行策略更新,依据奖励波动性和探索驱动的采集机制。DoPR将推演开销降低近一个数量级,同时保持竞争性推理准确率,为大模型后训练提供可扩展、资源高效的解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) have exhibited remarkable performance on complex reasoning tasks, with reinforcement learning under verifiable rewards (RLVR) emerging as a principled framework for aligning model behavior with reasoning chains. Despite its promise, RLVR remains prohibitively resource-intensive, requiring extensive reward signals and incurring substantial rollout costs during training. In this work, we revisit the fundamental question of data and compute efficiency in RLVR. We first establish a theoretical lower bound on the sample complexity required to unlock reasoning capabilities, and empirically validate that strong performance can be achieved with a surprisingly small number of training instances. To tackle the computational burden, we propose Dynamic One-Shot Policy Refinement (DoPR), an uncertainty-aware RL strategy that dynamically selects a single informative training sample per batch for policy updates, guided by reward volatility and exploration-driven acquisition. DoPR reduces rollout overhead by nearly an order of magnitude while preserving competitive reasoning accuracy, offering a scalable and resource-efficient solution for LLM post-training. This approach offers a practical path toward more efficient and accessible RL-based training for reasoning-intensive LLM applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。