让扩散模型记住长期上下文,解决生成画面失真的问题。
StateSpaceDiffuser: Bringing Long Context to Diffusion World Models
- 用状态空间模型融合历史交互信息,增强扩散模型的长时记忆。
- 在2D迷宫和3D环境中,生成结果可保持一致性长达十倍以上步数。
- 适合需要长时间视觉连贯性的场景预测任务,如机器人导航。
世界模型近年来在复杂环境中的动作条件视觉预测中备受关注。然而,仅依赖少量近期观测会导致其丢失长期上下文,使生成场景在几步内偏离先前观察内容,破坏时间连贯性。这一限制普遍存在于当前基于扩散模型的世界模型中,根源在于缺乏持久的环境状态表示。为此,我们提出StateSpaceDiffuser,通过将状态空间模型提取的特征融入扩散模型,使其具备长上下文处理能力。该设计在保留扩散模型高保真合成能力的同时恢复长期记忆。为严格评估时间一致性,我们开发了测试协议,检验模型在延长推演中重现已见内容的能力。大量实验表明,StateSpaceDiffuser显著优于强基准的纯扩散模型,在2D迷宫与复杂3D环境中均能维持更长时间的视觉连贯性,实现了视觉细节与长期记忆的协同提升。
原文摘要 · Abstract (English)
World models have recently gained prominence for action-conditioned visual prediction in complex environments. However, relying on only a few recent observations causes them to lose long-term context. Consequently, within a few steps, the generated scenes drift from what was previously observed, undermining temporal coherence. This limitation, common in state-of-the-art world models, which are diffusion-based, stems from the lack of a lasting environment state. To address this problem, we introduce StateSpaceDiffuser, where a diffusion model is enabled to perform long-context tasks by integrating features from a state-space model, representing the entire interaction history. This design restores long-term memory while preserving the high-fidelity synthesis of diffusion models. To rigorously measure temporal consistency, we develop an evaluation protocol that probes a model's ability to reinstantiate seen content in extended rollouts. Comprehensive experiments show that StateSpaceDiffuser significantly outperforms a strong diffusion-only baseline, maintaining a coherent visual context for an order of magnitude more steps. It delivers consistent views in both a 2D maze navigation and a complex 3D environment. These results establish that bringing state-space representations into diffusion models is highly effective in demonstrating both visual details and long-term memory. Project page: https://insait-institute.github.io/StateSpaceDiffuser/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。