为安全关键的智能体系统设计能识别重要后果的世界模型。
Rethinking World Models for Safety-Critical Embodied Systems

- 以决策为中心构建世界模型,关注后果与不确定性。
- 整合反事实推理与安全记忆,支持实时风险评估。
- 适合自动驾驶、机器人等高风险场景研究者。
世界模型已从紧凑的隐空间动态发展为生成式、可控且可交互的具身环境模拟器。然而,高预测似然性和视觉保真度并不保证模型能保留安全决策所需的证据。本文指出当前世界建模存在三大结构性错配:似然性与风险、预测与干预、有限时域预测与累积后果。为此提出风险感知世界模型(RIWM),围绕后果、干预、认知不确定性与可恢复性组织建模,集成四项互相关联能力:决策相关表征、反事实推理、安全关键的情景记忆及运行时安全保障。模型区分物理、社会和操作后果,并利用认知不确定性量化行动依据。文章进一步讨论开放挑战:识别有影响的未来、验证反事实推理、维护可修订的安全记忆、将学习到的后果转化为可执行约束,以及判断证据是否足够行动。主张未来世界模型应超越预测可能未来,转向识别哪些未来重要、通过经验修正判断,并明确何时行动、修正、感知、延迟或放弃。
原文摘要 · Abstract (English)
World models have progressed from compact latent dynamics to generative, controllable, and interactive simulators of embodied environments. However, high predictive likelihood and visual fidelity do not necessarily ensure that a model preserves the evidence required for safe decision-making. This perspective identifies three structural mismatches in current world modeling: likelihood versus risk, prediction versus intervention, and finite-horizon prediction versus accumulated consequences. We propose the Risk-Informed World Model (RIWM) as a decision-centric research direction for safety-critical embodied systems. RIWM organizes world modeling around consequences, intervention, epistemic uncertainty, and recoverability, and integrates four interdependent capabilities: decision-relevant representation, counterfactual reasoning, safety-critical episodic memory, and runtime safety assurance. It distinguishes physical, social, and operational consequences while using epistemic uncertainty to qualify the evidence supporting action. We further discuss open challenges in identifying consequential futures, validating counterfactual reasoning, maintaining revisable safety memories, translating learned consequences into executable constraints, and determining when evidence is sufficient to act. This perspective argues that future world models should move beyond predicting likely futures toward identifying which futures matter, revising judgments through experience, and recognizing when to act, revise, sense, defer, or abstain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。