根据任务难易动态分配采样，提升强化学习样本效率。
STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization
- 按任务成功率动态调整采样，优先关注难任务
- 将轨迹分解为步骤级样本，提升更新精度
- 在相同采样量下收敛更快，泛化能力更强
多轮交互对在线强化学习仍是挑战。传统轨迹级优化将每条轨迹视为单一训练样本，存在采样效率低、学习信号误导等问题：对所有任务均匀采样、惩罚失败轨迹中的正确中间动作、采样成本高。为此，我们提出STEP（Success-rate-aware Trajectory-Efficient Policy optimization），基于任务成功率动态分配采样并进行步骤级优化。STEP维护平滑的成功率记录，引导自适应轨迹重采样，向更难任务倾斜资源；计算成功率加权优势，并将轨迹分解为步骤级样本；最后采用步骤级GRPO增强，优化低成功率任务的更新。在OSWorld和AndroidWorld上的实验表明，相比轨迹级GRPO，STEP在相同采样预算下显著提升样本效率与训练稳定性，收敛更快，泛化性能更好。
原文摘要 · Abstract (English)
Multi-turn interaction remains challenging for online reinforcement learning. A common solution is trajectory-level optimization, which treats each trajectory as a single training sample. However, this approach can be inefficient and yield misleading learning signals: it applies uniform sampling across tasks regardless of difficulty, penalizes correct intermediate actions in failed trajectories, and incurs high sample-collection costs. To address these issues, we propose STEP (Success-rate-aware Trajectory-Efficient Policy optimization), a framework that dynamically allocates sampling based on per-task success rates and performs step-level optimization. STEP maintains a smoothed success-rate record to guide adaptive trajectory resampling, allocating more effort to harder tasks. It then computes success-rate-weighted advantages and decomposes trajectories into step-level samples. Finally, it applies a step-level GRPO augmentation to refine updates for low-success tasks. Experiments on OSWorld and AndroidWorld show that STEP substantially improves sample efficiency and training stability over trajectory-level GRPO, converging faster and generalizing better under the same sampling budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。