arXiv:2607.15142cs.AIcs.LG2026-07

为提升游戏世界模型可靠性,提出概念引导的空间正则化方法

Concept-Guided Spatial Regularization for World Models in Atari Pong

论文配图:Concept-Guided Spatial Regularization for World Models in Atari Pong
图 1 · 摘自论文原文
  • 在冻结的世界模型上引入概念分割与空间正则化损失
  • 使模型生成更准确的球轨迹,零样本强化学习性能提升显著
  • 适合关注模型可信赖性与世界建模泛化能力的研究者

世界模型通常作为基于模型的强化学习(MBRL)系统的一部分进行评估,其独立可靠性未被充分研究。我们复现了Atari Pong中的五种视觉世界模型代理——DreamerV3、DIAMOND、TWISTER、Simulus和STORM,并匹配其报告的代理性能。随后冻结已学世界模型,通过两种方式评估:一是闭环滚动诊断,用独立训练的策略与冻结模型交互,观察生成视觉轨迹中的视觉与动力学错误;二是像素空间零样本MBRL,新策略完全在冻结模型内训练并评估于真实环境。结果发现所有模型均存在明显失败,如球消失、运动错误、球拍交互异常;零样本策略性能显著低于原生MBRL管道。以DreamerV3为例,平均回报从-5.5降至-20.9,接近最低值-21。我们推测任务关键概念(如球)建模不足是主因,提出概念引导的空间正则化(CGSReg),在分割概念区域上添加辅助重建损失。该方法在DreamerV3、DIAMOND、TWISTER中改善了闭环滚动与零样本MBRL,在Simulus中改善了零样本性能,但STORM未见明显提升。

原文摘要 · Abstract (English)

World models are usually evaluated as components of model-based reinforcement learning (MBRL) systems, leaving their standalone reliability understudied. We reproduce five visual world-model agents in Atari Pong -- DreamerV3, DIAMOND, TWISTER, Simulus, and STORM -- and match their reported agent performance. We then freeze the learned world models and evaluate them in two ways. In a closed-loop rollout diagnostic, a policy trained separately from the corresponding MBRL agent interacts with each frozen model, and we inspect the generated visual trajectories for visual and dynamical errors. Across all five models, these rollouts contain clear failures, including ball disappearance, incorrect motion, and invalid ball-paddle interactions. Beyond visual trajectories, we further evaluate the frozen models with pixel-space zero-shot MBRL, a challenging setting in which a new policy is trained entirely inside each frozen world model and then evaluated in the real environment. Across all five models, these policies substantially underperform those produced by the corresponding original MBRL pipelines. For DreamerV3, mean return drops from $-5.5$ to $-20.9$, near the minimum of $-21$. We hypothesize that insufficient modeling of task-critical concepts, such as the ball in Pong, contributes to these failures and propose Concept-Guided Spatial Regularization (CGSReg), an auxiliary reconstruction loss on segmented concept regions. CGSReg improves both closed-loop rollouts and pixel-space zero-shot MBRL in DreamerV3, DIAMOND, and TWISTER, and improves zero-shot MBRL in Simulus; STORM shows no clear improvement.

世界模型强化学习概念建模游戏模拟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。