arXiv:2502.11537cs.LGcs.AI2025-02被引 3

Simulus整合多项改进,让世界模型在少样本下更强更稳。

Simulus: Combining Improvements in Sample-Efficient World Model Agents

  • 模块化设计支持多模态输入与灵活组合
  • 在三个基准上达少样本规划类世界模型最优表现
  • 适合研究样本效率强化学习的开发者参考

世界模型(WMs)代表了少样本强化学习的前沿,但其复杂性使得许多已知改进难以应用,因需大量专业知识和投入。受Rainbow启发——它证明了对DQN的多项改进可协同增效,我们探索这一原则是否适用于世界模型代理。提出Simulus,一种基于标记的模块化WM代理,集成:(1) 支持任意观测与动作模态组合的灵活标记化框架;(2) 用于减少认知不确定性的内在动机;(3) 优先级世界模型重放;(4) 奖励与回报预测的回归转分类方法。Simulus在三个多样化基准上实现无规划世界模型的最先进样本效率:视觉Atari 100K、连续控制DMC Proprioception 500K、符号型Craftax-1M。值得注意的是,即使在极有限的交互预算下,内在动机仍具价值,尽管存在将稀缺交互浪费于无关经验的风险。消融实验表明各组件独立有效,组合产生协同增益。代码与模型权重已公开于https://github.com/leor-c/Simulus。

原文摘要 · Abstract (English)

World models (WMs) represent the frontier of sample-efficient reinforcement learning, but their complexity leaves many promising improvements unrealized due to the significant expertise and effort required to identify and integrate them. Inspired by Rainbow, which showed that individually known improvements to DQN complement each other and can be effectively combined, we take on this challenge and ask whether the same principle applies to world model agents. We introduce Simulus, a modular token-based WM agent that integrates: (1) a flexible tokenization framework supporting arbitrary combinations of observation and action modalities; (2) intrinsic motivation for epistemic uncertainty reduction; (3) prioritized world model replay; and (4) regression-as-classification for reward and return prediction. Simulus achieves state-of-the-art sample efficiency for planning-free WMs across three diverse benchmarks: visual Atari 100K, continuous-control DMC Proprioception 500K, and symbolic Craftax-1M. Notably, intrinsic motivation proves beneficial even under the tight interaction budgets of sample-efficient RL, despite the risk of wasting scarce interactions on task-irrelevant experience. Ablation studies reveal that each component contributes individually, and their combination yields synergistic gains. Our code and model weights are publicly available at https://github.com/leor-c/Simulus.

世界模型少样本学习强化学习模块化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。