用深度图构建轻量3D记忆，提升长视频生成的几何一致性。
Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generation
- 基于预训练3D先验生成目标视角覆盖图作为记忆证据
- 迭代选择能最大化未覆盖区域的帧，提升记忆覆盖率
- 滑动窗口缓存深度图，支持长视频高效生成
长时序自回归视频生成中保持几何一致性仍具挑战。现有记忆增强生成模型依赖历史帧检索，但其效果受两个关键设计影响：如何表示过去的3D几何证据，以及如何从这些证据中选择记忆帧。现有方法多依赖相机位姿或视场重叠，虽轻量但粗粒度，难以推断像素级可见性；或采用显式3D重建，虽精细却难以长期维护。本文提出基于深度图的记忆检索框架COVRAG，利用预训练3D先验构建目标视角覆盖图作为轻量3D记忆证据。在帧选择上，COVRAG最大化残余覆盖增益，迭代检索当前上下文与已选记忆未覆盖的目标视角区域。为提升长视频生成可扩展性，引入滑动窗口深度缓存以高效估计几何信息。在RealEstate10K和DL3DV10K数据集上的实验表明，COVRAG在保持低延迟的同时显著提升了长时序几何一致性。
原文摘要 · Abstract (English)
Maintaining long-term geometric consistency remains challenging for long-horizon autoregressive video generation. Memory-augmented generative models address this by retrieving historical frames, but their effectiveness depends on two key design choices: what 3D-geometric evidence should represent past observations, and how memory frames should be selected from this evidence. Existing methods often rely on camera poses or field-of-view overlap, which are lightweight but too coarse to reason about pixel-wise visibility, or use explicit 3D reconstruction, which provides fine-grained evidence but is costly to maintain over long rollouts. We propose Coverage-Maximizing Retrieval-Augmented Generation (COVRAG), a depth-based memory retrieval framework that uses pretrained 3D priors to construct a target-view coverage map as lightweight 3D memory evidence. For frame selection, COVRAG maximizes residual coverage gain, iteratively retrieving frames that explain target-view regions not covered by the current context or previously selected memories. To improve scalability in long-video generation, we introduce sliding-window depth caching for efficient geometry estimation. Experiments on RealEstate10K and DL3DV10K show that COVRAG improves long-horizon geometric consistency while maintaining low latency compared to baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。