不用解码器和数据增强，用内部正则化提升世界模型性能
R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation
- 设计冗余削减目标，替代外部数据增强进行自监督训练
- 在DMC和Meta-World上表现媲美DreamerV3，训练快1.59倍
- 特别适合小目标任务，对视觉细节不敏感
基于图像的模型基础强化学习（MBRL）的核心挑战在于从无关视觉细节中提炼关键信息。重建类方法常浪费资源于任务无关区域。无解码器方法依赖数据增强（DA）学习鲁棒表示，但其外部正则化限制了泛化能力。本文提出R2-Dreamer，一种无需解码器或增强的MBRL框架，采用自监督目标作为内部正则化，防止表征崩溃。核心是受Barlow Twins启发的冗余削减目标，可轻松集成至现有框架。在DeepMind Control Suite和Meta-World上，R2-Dreamer性能与DreamerV3、TD-MPC2相当，训练速度比DreamerV3快1.59倍；在包含微小任务相关物体的DMC-Subtle上表现显著优于基线。结果表明，有效的内部正则化可实现灵活且高性能的无解码器MBRL。代码已开源。
原文摘要 · Abstract (English)
A central challenge in image-based Model-Based Reinforcement Learning (MBRL) is to learn representations that distill essential information from irrelevant visual details. While promising, reconstruction-based methods often waste capacity on large task-irrelevant regions. Decoder-free methods instead learn robust representations by leveraging Data Augmentation (DA), but reliance on such external regularizers limits versatility. We propose R2-Dreamer, a decoder-free MBRL framework with a self-supervised objective that serves as an internal regularizer, preventing representation collapse without resorting to DA. The core of our method is a redundancy-reduction objective inspired by Barlow Twins, which can be easily integrated into existing frameworks. On DeepMind Control Suite and Meta-World, R2-Dreamer is competitive with strong baselines such as DreamerV3 and TD-MPC2 while training 1.59x faster than DreamerV3, and yields substantial gains on DMC-Subtle with tiny task-relevant objects. These results suggest that an effective internal regularizer can enable versatile, high-performance decoder-free MBRL. Code is available at https://github.com/NM512/r2dreamer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。