用自动停止机制让先验数据训练更省算力且不超调。
SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data
- 基于策略对齐的离线评估信号动态决定训练时长。
- 在25个连续控制任务上性能提升最高45.6%，算力减少22倍。
- 适合追求高效在线强化学习的算法研发者。
将先验数据引入在线强化学习可加速训练，但通常面临高计算成本与长多阶段训练流程之间的权衡。尽管固定长度的稳定阶段比静态更新策略更高效，却需依赖任务相关的手动调参，可能导致先验知识浪费或严重过拟合。为此，我们提出SOPE，利用与策略对齐的离线策略评估（OPE）信号作为自动化早停机制，动态控制离线训练阶段长度。通过在保留验证集上以当前策略的动作分布评估价值函数，SOPE在分布外收益饱和时精确终止梯度更新，无需手动调度。在Minari基准套件的25个连续控制任务上，SOPE将基线性能提升高达45.6%，同时减少高达22倍的所需TFLOPs，有效平衡了样本与计算效率。结果表明，自适应、评估驱动的更新调度优于依赖静态、耗时的调度方案。
原文摘要 · Abstract (English)
Incorporating prior data into online reinforcement learning accelerates training but typically forces a difficult trade-off between high computational costs and long, multi-stage training pipelines. While fixed-length stabilization phases are significantly more computationally efficient than static update schedules, they require task-dependent manual tuning, risking either the waste of prior knowledge or severe overfitting. To address this, we propose SOPE, an algorithm that uses an actor-aligned Off-Policy Policy Evaluation (OPE) signal as an automated early-stopping mechanism to dynamically control the length of offline training phases. By evaluating the critic on a held-out validation split under the current policy's action distribution, SOPE halts gradient updates exactly when out-of-distribution benefits saturate, eliminating the need for manual schedule tuning. Evaluated on 25 continuous control tasks from the Minari benchmark suite, SOPE improves baseline performance by up to 45.6% while reducing the required TFLOPs by up to 22x, thus balancing the tradeoff between sample and computational efficiency. These findings demonstrate that adaptive, evaluation-driven update schedules are more effective than relying on static, exhaustive update schedules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。