用单次采样提升大模型推理，靠批量信息共享实现高效价值估计
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
- 单提示仅采样一次，但利用批量中跨提示信息优化价值估计
- 相比基线降低69%的值函数误差，用1次采样胜过8次采样的群均值方法
- 训练时间更少，性能接近多轮次算法，适合追求效率的推理优化场景
基于可验证奖励的强化学习已成为提升大语言模型推理能力的标准方法。现有算法在价值估计与策略学习中面临计算效率与样本效率的权衡。本文提出BASIS，一种无需评判器的后训练算法，旨在解决这一矛盾。每个在线训练步骤中，BASIS对每个提示仅采样一次，但通过利用整个批次中跨提示的丰富信息来改进价值函数估计。实验表明，与代表性单次采样基线REINFORCE++相比，BASIS将值函数估计的均方误差降低69%；且仅用一次采样，其误差低于使用8次采样的群均值估计器。该价值估计的提升带来了更好的策略优化：在显著减少训练时间的前提下，BASIS性能接近多轮次GRPO类方法，并经常超越单次采样REINFORCE类方法。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards has become a standard recipe for improving the reasoning abilities of large language models. Existing algorithms face a tradeoff between computational efficiency and sample efficiency in value estimation and policy learning. We introduce BASIS, a critic-free post-training algorithm designed to address this tradeoff. At each online training step, BASIS samples only one rollout per prompt, but leverages rich information across prompts in the entire batch to improve value function estimation. Our experiments demonstrate that BASIS reduces MSE in value function estimation by 69% compared to REINFORCE++, a representative single-rollout baseline, and achieves lower MSE with one rollout than group mean estimators with 8 rollouts. This improvement in value estimation translates to better policy optimization: using substantially less training time, BASIS achieves performance close to multi-rollout GRPO-type baselines and often outperforms single-rollout REINFORCE-type baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。