arXiv:2607.17326cs.LGcs.RO2026-07

实测发现:训练时间比样本效率更关键,域随机化对各类强化学习算法效果相似。

Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraints

论文配图:Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraints
图 1 · 摘自论文原文
  • 对比算法在真实时间(墙钟时间)上的训练速度,而非仅看样本效率。
  • 样本效率低的PPO反而比SAC、TD-MPC2更快产出可用策略。
  • 域随机化能有效提升不同算法的鲁棒性,适用性广泛。

面向迁移的强化学习需超越传统样本效率的评估维度。本文关注两个维度:实际效率——在墙钟时间而非交互步数限制下,算法适用性是否变化;以及动态不匹配下的鲁棒性——域随机化带来的训练分布变化如何影响不同学习范式。我们得出两点启示:首先,在迁移场景中,仅比较样本效率不足以判断算法适用性。实际训练时间至关重要,实验显示,样本效率较低的PPO在并行训练下比SAC和TD-MPC2更快生成性能良好的策略。其次,域随机化能有效帮助不同范式的算法学习鲁棒策略。尽管PPO(在线)、SAC(离线)、TD-MPC2(基于模型)代表不同学习范式,但域随机化对其影响方式相似。这是首次在相同迁移协议下,对三类算法在域随机化覆盖下的系统性对比。整体表明,评估强化学习算法应兼顾训练时间与生成可用策略的能力。

原文摘要 · Abstract (English)

Transfer-oriented reinforcement learning requires evaluating algorithms along dimensions that go beyond standard sample efficiency. We focus on two dimensions: practical efficiency, which asks whether conclusions about algorithm suitability change under wall-clock rather than interaction-based budgets, and robustness under dynamics mismatch, which asks how different learning paradigms respond to variability in the training distribution induced by domain randomization. We provide two insights to reinforcement-learning practitioners. First, comparing the sample efficiency of different algorithms is often an insufficient criterion in transfer-oriented settings. The wall-clock time required to train a decent policy is an important consideration for practitioners, and we find that the sample-inefficient PPO algorithm can produce a performant policy faster than relatively more sample-efficient algorithms such as SAC and TD-MPC2, validating the common understanding of massively parallel training paradigms. Second, domain randomization can help different kinds of algorithms learn robust policies. In particular, although PPO, SAC, and TD-MPC2 represent different RL paradigms - on-policy, off-policy, and model-based learning and planning, respectively - we find that domain randomization affects all three algorithms in a similar way. To the best of our knowledge, this is the first controlled comparison of the effect of domain-randomization coverage on PPO, SAC, and TD-MPC2 under the same transfer protocol. Taken together, these two insights highlight the importance of evaluating RL algorithms not only by sample efficiency, but also by practical considerations such as training time and the algorithms' ability to produce usable policies.

强化学习迁移学习鲁棒性训练效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。