用少量示范引导强化学习，让模型学会自主推理。
Context Bootstrapped Reinforcement Learning
- 在提示中随机添加少量示范，逐步减少依赖以促进内化
- 在五个推理任务上成功提升成功率，探索效率显著改善
- 适用于新语言和复杂推理，对模型架构无依赖
基于可验证奖励的强化学习（RLVR）面临探索效率低的问题，模型难以生成有效轨迹，导致学习信号微弱。尤其在需要掌握新推理模式或领域知识的任务中更为严重。为此，我们提出上下文自举强化学习（CBRL），通过随机在训练提示前添加少量少样本示范来增强训练。示范注入概率按课程学习策略从高逐渐降至零，迫使模型最终无需依赖示范即可成功，从而将示范中的推理模式内化为自身能力。我们在两个模型族和五个推理游戏任务上验证了CBRL，结果表明其能持续提升成功率、改善探索效率，且与算法无关。进一步实验显示，该方法在Q这一与主流编程语言差异显著的领域特定语言上也具实际应用价值。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) suffers from exploration inefficiency, where models struggle to generate successful rollouts, resulting in minimal learning signal. This challenge is particularly severe for tasks that require the acquisition of novel reasoning patterns or domain-specific knowledge. To address this, we propose Context Bootstrapped Reinforcement Learning (CBRL), which augments RLVR training by stochastically prepending few-shot demonstrations to training prompts. The injection probability follows a curriculum that starts high to bootstrap early exploration, then anneals to zero so the model must ultimately succeed without assistance. This forces the policy to internalize reasoning patterns from the demonstrations rather than relying on them at test time. We validate CBRL across two model families and five Reasoning Gym tasks. Our results demonstrate that CBRL consistently improves success rate, provides better exploration efficiency, and is algorithm-agnostic. We further demonstrate CBRL's practical applicability on Q, a domain-specific programming language that diverges significantly from mainstream language conventions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。