用视频模型完成稀疏视角合成，让模型忽略输入顺序。
FrameCrafter: Novel View Synthesis as Video Completion
- 将稀疏视角合成视为低帧率视频补全任务，利用视频模型隐含多视角知识。
- 仅需约5张多视角图像和相机位姿，即可生成目标视角，性能媲美主流方法。
- 通过去除时间位置编码等设计，使模型对输入顺序无感，适合任意顺序输入。
我们研究使用视频扩散模型解决稀疏视角合成（NVS）问题：给定约5张场景的多视角图像及其相机位姿，预测目标相机位姿下的视图。以往方法依赖单图生成先验，缺乏多视角信息。我们提出视频模型天然包含多视角知识，更适合用于NVS。核心思想是将稀疏NVS建模为低帧率视频补全任务。但稀疏输入通常是无序集合，难以定义合理顺序，因此模型需对输入排列保持不变。为此，我们提出FrameCrafter，通过引入每帧潜在编码并移除时间位置嵌入等结构修改，使视频模型适应排列不变的NVS任务。实验表明，视频模型可在极少量监督下“遗忘”时间顺序，实现与当前主流方法相当的稀疏视角合成性能。
原文摘要 · Abstract (English)
We tackle the problem of sparse novel view synthesis (NVS) using video diffusion models; given $K$ ($\approx 5$) multi-view images of a scene and their camera poses, we predict the view from a target camera pose. Many prior approaches leverage generative image priors encoded via diffusion models. However, models trained on single images lack multi-view knowledge. We instead argue that video models already contain implicit multi-view knowledge and so should be easier to adapt for NVS. Our key insight is to formulate sparse NVS as a low frame-rate video completion task. However, one challenge is that sparse NVS is defined over an unordered set of inputs, often too sparse to admit a meaningful order, so the models should be $\textit{invariant}$ to permutations of that input set. To this end, we present FrameCrafter, which adapts video models (naturally trained with coherent frame orderings) to permutation-invariant NVS through several architectural modifications, including per-frame latent encodings and removal of temporal positional embeddings. Our results suggest that video models can be easily trained to "forget" about time with minimal supervision, producing competitive performance on sparse-view NVS benchmarks. Project page: https://frame-crafter.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。