无需标签和参考模型，用离线强化学习让大模型高效自进化推理。
RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
- 基于离线迭代强化学习，优化加权对数似然目标。
- 训练速度达TTRL的2.5倍，且在推理任务上表现更优。
- 适合追求高效自训练的大模型研究与应用落地。
强化学习（RL）是提升大语言模型（LLMs）推理能力的核心方法，但通常依赖真实奖励信号。测试时强化学习（TTRL）通过多数投票机制避免了这一需求，但需大量在线强化学习，计算成本高昂。本文提出RoiRL：一种面向离线迭代强化学习的轻量级替代方案，可逼近相同的正则化最优策略。与TTRL不同，RoiRL无需维护参考模型，转而优化加权对数似然目标，实现更稳定的训练，显著降低内存与计算开销。实验表明，RoiRL训练速度比TTRL快2.5倍，并在多个推理基准上持续超越其性能，为无标注自提升大模型提供了可扩展路径。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is central to improving reasoning in large language models (LLMs) but typically requires ground-truth rewards. Test-Time Reinforcement Learning (TTRL) removes this need by using majority-vote rewards, but relies on heavy online RL and incurs substantial computational cost. We propose RoiRL: Reasoning with offline iterative Reinforcement Learning, a family of lightweight offline learning alternatives that can target the same regularized optimal policies. Unlike TTRL, RoiRL eliminates the need to maintain a reference model and instead optimizes weighted log-likelihood objectives, enabling stable training with significantly lower memory and compute requirements. Experimental results show that RoiRL trains to 2.5x faster and consistently outperforms TTRL on reasoning benchmarks, establishing a scalable path to self-improving LLMs without labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。