提出单次采样训练新方法，解决强化学习中分组采样的低效问题。
Rethinking Groups in Critic-Free RLVR
- 用负令牌过滤替代传统分组，实现单次采样稳定训练
- 在推理任务上性能相当，在智能体任务上优于传统分组方法
- 适合追求高效、灵活的RL训练场景
强化学习已成为大语言模型后训练的核心范式。现有无评价者（critic-free）RL方法通常对同一问题生成一组轨迹以估计价值基线，用于优势计算。然而，该设计存在数据效率低、分组同步障碍以及对结构化轨迹适应性差的问题。本文重新审视‘分组’的作用，发现其核心功能并非仅估计基线，而是防止对负样本产生误罚。基于此洞察，我们提出负令牌过滤策略，一种简单有效的机制，支持稳定单轨迹训练。将其应用于两种批处理级优势计算方法，在推理任务上达到相当性能，在智能体任务上表现更优，超越传统分组式RL技术。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a central paradigm for post-training large language models. Existing critic-free RL methods typically generate a group of rollouts for the same question to estimate value baselines for advantage computation. However, this design suffers from data inefficiency, group synchronization barriers, and inflexibility with structured rollouts. In this work, we revisit the role of the ``group'' and show that its underlying function is not merely to estimate baselines but to prevent false penalties on negative samples. Building on this insight, we propose negative token filtering, a simple and effective strategy that enables stable single-rollout training. We apply it to two batch-level advantage methods, achieving comparable performance on reasoning tasks and stronger performance on agentic tasks relative to group-based RL techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。