arXiv:2606.09828cs.CV2026-06被引 3

用扩散模型隐空间构建3D场景记忆,加速视频生成并减少内存占用。

Latent Spatial Memory for Video World Models

论文配图:Latent Spatial Memory for Video World Models
图 1 · 摘自论文原文
  • 将场景信息直接存于扩散模型的隐空间,避免像素空间重建
  • 端到端生成速度提升10.57倍,内存降低55倍
  • 适合需要高效视频生成与3D一致性的研究者

维持生成帧间3D空间一致性的视频世界模型通常依赖于在RGB空间中构建的显式点云记忆。这种设计既计算成本高,需反复渲染和VAE编码,又固有信息损失,因往返像素空间会丢弃学习到的隐表示丰富特征。本文提出「隐空间3D记忆」,一种持久的3D缓存,直接在扩散模型隐空间中存储场景信息,避免像素空间重构。基于此,我们提出Mirage框架:通过深度引导反投影将隐令牌升维至3D,并通过直接隐空间变形合成新视角进行查询。该统一方法同时消除像素空间重构的信息损失和重复编码渲染的计算负担。实验表明,隐空间记忆实现最高10.57倍的端到端视频生成加速,相比显式3D基线内存减少55倍。利用扩散模型的几何先验,Mirage在WorldScore上达到最先进性能,在RealEstate10K上展现强重建质量。

原文摘要 · Abstract (English)

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce \emph{latent spatial memory} for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to \textbf{10.57}$\times$ faster end-to-end video generation and \textbf{55}$\times$ reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.

视频生成隐空间3D一致扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。