通过减去动作均值,让世界模型自动过滤无关运动干扰。
Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection

- 用动作均值减法分离出可控动作信号
- 在多种场景下显著提升对自身动作的识别能力
- 无需额外训练,可直接应用于现有模型
潜在世界模型通过预测动作引发的未来状态进行规划,但当场景中存在非控制运动时,不同动作的预测会变得难以区分,尽管训练损失仍在下降。现有方法通过重建、任务奖励或辅助目标来缓解此问题,但增加了复杂性。本文提出一种极简替代方案:借鉴价值分解中状态基线与动作优势的分离思想,在潜在动态中减去动作均值,从而消除所有动作共有的非控制变化(即干扰源),保留纯净的可控动作通道。该操作仅在读取时执行,适用于任意动作条件的世界模型,包括冻结的预训练模型。在网格世界、带已知因子的合成生成器、连续控制干扰和自然像素的Atari游戏上,该方法使原本被混淆的预测恢复了对自身动作的敏感性,干扰泄漏几乎为零;作为后处理应用,它能从现成模型中提取出原始输出所遗漏的动作通道,并将网格世界模型转化为目标导向控制。我们证明该抵消在有限样本下对离散和采样动作集均为精确成立,并在附录中给出其有效边界——当干扰运动与动作同步时失效,同时说明剩余局限性。
原文摘要 · Abstract (English)
Latent world models plan by predicting future states from an action, but when a scene contains motion the agent does not control, they quietly go action-blind: predictions for different actions become indistinguishable even as the training loss keeps improving. Existing remedies suppress this distraction with reconstruction, task reward, or auxiliary objectives, each adding machinery or assumptions. We show that a minimal alternative suffices, borrowed from the dueling decomposition of value into a state baseline and an action advantage: in latent dynamics, subtracting a prediction's mean effect over actions cancels whatever the actions share--the action-independent variation where distractors live--leaving a clean, controllable channel, with no reward, no reconstruction, and no distractor-specific auxiliary loss. Because this is only a subtraction at readout time, it applies unchanged to any action-conditioned world model, including frozen pretrained ones. Across a gridworld, synthetic generators with known factors, distracting continuous control, and natural-pixel Atari, the isolated channel recovers the agent's own effect where entangled predictors fail, with nuisance leak indistinguishable from zero; applied post hoc it surfaces an action channel in off-the-shelf models that their raw readouts miss, and it converts into goal-reaching control in the gridworld. We prove the cancellation is exact in finite samples for both discrete and sampled action sets, and we state its measured boundary--distractors whose motion tracks the action--together with the remaining limitations in the appendix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。