无需训练即可解决3D生成视频重访不一致问题
Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering
- 利用3D引擎提供的时序与空间对应关系，回溯历史隐状态作为闭环记忆
- 在TartanAir和TartanGround数据集上，重访一致性显著优于现有无训练基线
- 适用于游戏和沉浸式内容生成等需长期连续生成的场景
近期条件视频生成模型展现出将3D引擎输出的深度图和未贴图几何体转化为逼真视频的潜力，应用于游戏与沉浸式内容创作。此类应用需长时程自回归生成，持续合成新帧的同时保持三维世界一致性。自回归生成器以有限缓存逐块合成视频，当相机在上下文被替换后重访某位置时，模型常产生不一致外观，尽管条件输入（如深度图）仍与底层几何对齐。本文提出无需训练的解决方案：利用3D引擎已提供的时序对应关系，将姿态匹配的历史隐状态块引入KV缓存作为闭环记忆；同时通过相机位姿与深度重投影的空间对应，引导注意力聚焦于重建块中几何对应的区域。我们在TartanAir和TartanGround数据集中挖掘环路轨迹，模拟复杂真实场景，结果表明该方法在保持整体视频质量前提下，显著提升重访一致性，优于现有无训练基线。
原文摘要 · Abstract (English)
Recent conditional video generation models have shown promising potentials to transform 3D engine renderings, such as depth maps and untextured geometry, into photorealistic videos for gaming and immersive content creation. These applications require long-horizon auto-regressive generation that continuously synthesizes new frames while preserving a persistent 3D world. Auto-regressive generators synthesize video chunk by chunk with a bounded KV cache, so when the camera revisits a location after its context has been evicted, the model often regenerates inconsistent appearance, even though the conditioning renderings (e.g., depth) remain perfectly aligned with the underlying geometry. We address this revisit inconsistency without any post-training by exploiting correspondences the 3D engine already provides: temporal correspondence retrieves pose-matched historical latent chunks into the KV cache as loop-closure memory, while spatial correspondence from camera pose and depth reprojection biases token-level attention toward geometrically corresponding regions of the retrieved chunks. We demonstrate our method on loop-closure trajectories mined from TartanAir and TartanGround dataset to mirror complicate real-world application scenarios, where it outperforms existing training-free baselines on revisit consistency without losing overall video quality. Project Page: https://wenchao-m.github.io/ClosetheLoop.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。