用乐观模型提升强化学习探索效率，无需不确定性估计。
Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning
- 在模型学习中加入乐观动态损失，引导想象轨迹向高奖励方向
- 在稀疏奖励环境下样本效率提升，累积回报显著优于基线
- 可直接嵌入现有世界模型，无需修改训练流程
高效探索仍是强化学习的核心挑战，尤其在奖励稀疏的环境中。本文提出乐观世界模型（OWMs），将自适应控制中的奖励偏置最大似然估计（RBMLE）引入深度强化学习，构建了一个原理清晰且可扩展的乐观探索框架。与基于上置信界（UCB）的方法不同，OWMs通过引入乐观动态损失，直接在模型学习中融入乐观性，使想象的环境转移倾向于更高奖励结果。该损失为全梯度化设计，无需不确定性估计或约束优化。本方法可无缝集成到现有世界模型框架中，仅需微小改动即可保持可扩展性。我们在两种前沿世界模型架构中实现了乐观版本：乐观DreamerV3和乐观STORM，实验表明其在样本效率和累积回报方面均显著优于原始模型。
原文摘要 · Abstract (English)
Efficient exploration remains a central challenge in reinforcement learning (RL), particularly in sparse-reward environments. We introduce Optimistic World Models (OWMs), a principled and scalable framework for optimistic exploration that brings classical reward-biased maximum likelihood estimation (RBMLE) from adaptive control into deep RL. In contrast to upper confidence bound (UCB)-style exploration methods, OWMs incorporate optimism directly into model learning by augmentation with an optimistic dynamics loss that biases imagined transitions toward higher-reward outcomes. This fully gradient-based loss requires neither uncertainty estimates nor constrained optimization. Our approach is plug-and-play with existing world model frameworks, preserving scalability while requiring only minimal modifications to standard training procedures. We instantiate OWMs within two state-of-the-art world model architectures, leading to Optimistic DreamerV3 and Optimistic STORM, which demonstrate significant improvements in sample efficiency and cumulative return compared to their baseline counterparts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。