用128条随机演示提升强化学习的样本效率，解决难任务训练瓶颈。
Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance
- 用少量随机选的演示数据引导强化学习，减少对大规模标注的依赖。
- 仅需128条演示即在多个任务上超越基线，接近全量数据效果。
- 结合监督信号、在线策略信号和权重衰减，防止过拟合，适合资源受限场景。
基于可验证奖励的强化学习（RLVR）在数学与编程等任务中通过思维链推理取得了显著进展。然而，在正确推理路径难以生成的难题上，其样本效率仍不足。现有方法采用示范引导的RLVR，即在强化学习失败时进行监督微调（SFT），但SFT通常需要大量数据，获取成本高。本文提出FEST算法，一种基于少量示范的RLVR方法。仅需从SFT数据集中随机选取128条演示即可取得优异表现。研究发现三个关键要素：监督信号、在线策略信号，以及对少量SFT数据使用递减权重以避免多轮训练带来的过拟合。在多个基准测试中，FEST以远少于基线所需的SFT数据量实现更优性能，甚至在全量数据下达到相当水平。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has achieved great success in developing Large Language Models (LLMs) with chain-of-thought rollouts for many tasks such as math and coding. Nevertheless, RLVR struggles with sample efficiency on difficult problems where correct rollouts are hard to generate. Prior works propose to address this issue via demonstration-guided RLVR, i.e., to conduct Supervised FineTuning (SFT) when RL fails; however, SFT often requires a lot of data, which can be expensive to acquire. In this paper, we propose FEST, a FEw-ShoT demonstration-guided RLVR algorithm. It attains compelling results with only 128 demonstrations randomly selected from an SFT dataset. We find that three components are vital for the success: supervised signal, on-policy signal, and decaying weights on the few-shot SFT dataset to prevent overfitting from multiple-epoch training. On several benchmarks, FEST outperforms baselines with magnitudes less SFT data, even matching their performance with full dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。