让强化学习模型像人一样识别事件,提升泛化与效率
From Observations to Events: Event-Aware World Model for Reinforcement Learning
- 通过自动识别事件边界,从原始观测中提取有意义的事件片段
- 在多个基准上使强基线模型性能提升10%-45%,达到新SOTA
- 无需人工标注,适用于多种复杂环境,适合做通用世界模型的研究者
基于模型的强化学习(MBRL)通过从原始观测中学习世界模型来提高样本效率,但现有方法在结构相似场景间泛化能力差,且易受纹理、颜色变化等无关干扰影响。从认知科学角度看,人类将连续感官流分割为离散事件,并以此决策。受此启发,我们提出事件感知世界模型(EAWM),一种无需人工标签的通用框架,可学习事件感知表示以简化策略学习。EAWM采用自动化事件生成器从原始观测中提取事件,并引入通用事件分割器(GES)识别事件边界,标记事件段的起止时间。通过事件预测,表征空间被塑造为捕捉有意义的时空转换。此外,我们提出了看似不同的世界模型架构的统一形式,并验证了方法的广泛适用性。在Atari 100K、Craftax 1M、DeepMind Control 500K和DMC-GB2 500K上的实验表明,EAWM始终使强基线模型性能提升10%-45%,在各基准上均达到新最优表现。代码已开源:https://github.com/MarquisDarwin/EAWM。
原文摘要 · Abstract (English)
While model-based reinforcement learning (MBRL) improves sample efficiency by learning world models from raw observations, existing methods struggle to generalize across structurally similar scenes and remain vulnerable to spurious variations such as textures or color shifts. From a cognitive science perspective, humans segment continuous sensory streams into discrete events and rely on these key events for decision-making. Motivated by this principle, we propose the Event-Aware World Model (EAWM), a general framework that learns event-aware representations to streamline policy learning without requiring handcrafted labels. EAWM employs an automated event generator to derive events from raw observations and introduces a Generic Event Segmentor (GES) to identify event boundaries, which mark the start and end time of event segments. Through event prediction, the representation space is shaped to capture meaningful spatio-temporal transitions. Beyond this, we present a unified formulation of seemingly distinct world model architectures and show the broad applicability of our methods. Experiments on Atari 100K, Craftax 1M, and DeepMind Control 500K, DMC-GB2 500K demonstrate that EAWM consistently boosts the performance of strong MBRL baselines by 10%-45%, setting new state-of-the-art results across benchmarks. Our code is released at https://github.com/MarquisDarwin/EAWM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。