发现基于模型的强化学习在不同环境表现差异大，合成数据反而拖后腿。
Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
- 用合成数据提升样本效率，是Dyna类算法的核心思路。
- 在DMC环境中，合成数据使性能显著下降，降幅达30%以上。
- 跨环境对比揭示：没有通用有效的强化学习优化方案。
Dyna-style离策略基于模型强化学习（DMBRL）通过生成合成状态转移数据来提升离策略RL的样本效率。本文研究发现，在具身感知观测下，尽管OpenAI Gym与DeepMind Control Suite（DMC）任务相似且物理引擎一致，但DMBRL在二者间表现差异显著：在DMC中性能大幅下降，即便采用现代改进技术也无法稳定提升。结果表明，将合成轨迹引入训练过程——这正是Dyna类方法的核心——反而在多数DMC环境中严重损害性能。该发现揭示了模型化强化学习中的根本挑战，印证了强化学习中‘无免费午餐’的规律：不同基准间的性能表现不可简单外推。
原文摘要 · Abstract (English)
Dyna-style off-policy model-based reinforcement learning (DMBRL) algorithms are a family of techniques for generating synthetic state transition data and thereby enhancing the sample efficiency of off-policy RL algorithms. This paper identifies and investigates a surprising performance gap observed when applying DMBRL algorithms across different benchmark environments with proprioceptive observations. We show that, while DMBRL algorithms perform well in OpenAI Gym, their performance can drop significantly in DeepMind Control Suite (DMC), even though these settings offer similar tasks and identical physics backends. Modern techniques designed to address several key issues that arise in these settings do not provide a consistent improvement across all environments, and overall our results show that adding synthetic rollouts to the training process -- the backbone of Dyna-style algorithms -- significantly degrades performance across most DMC environments. Our findings contribute to a deeper understanding of several fundamental challenges in model-based RL and show that, like many optimization fields, there is no free lunch when evaluating performance across diverse benchmarks in RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。