用语义特征提升生成新视角的准确性和一致性。
SemanticNVS: Improving Semantic Scene Understanding in Generative Novel View Synthesis
- 引入预训练语义提取器作为条件，增强场景理解。
- 在多个数据集上实现FID降低4.69%至15.26%。
- 适合关注高质量长距离视角生成的研究者。
我们提出SemanticNVS，一种基于相机条件的多视角扩散模型，用于新视角合成（NVS），通过整合预训练语义特征提取器来提升生成质量和一致性。现有NVS方法在靠近输入视角时表现良好，但在远距离相机运动下常生成语义不合理且失真的图像，表现出严重退化。我们推测这源于当前模型未能充分理解其条件或中间生成的场景内容。为此，我们提出集成预训练语义特征提取器，以更强的场景语义作为条件，实现在远距离视角下的高质量生成。我们研究了两种策略：(1) 透视变形的语义特征；(2) 在每个去噪步骤中交替进行理解与生成。在多个数据集上的实验表明，相比现有最优方法，本方法在定性和定量上均有明显提升，FID降低4.69%至15.26%。
原文摘要 · Abstract (English)
We present SemanticNVS, a camera-conditioned multi-view diffusion model for novel view synthesis (NVS), which improves generation quality and consistency by integrating pre-trained semantic feature extractors. Existing NVS methods perform well for views near the input view, however, they tend to generate semantically implausible and distorted images under long-range camera motion, revealing severe degradation. We speculate that this degradation is due to current models failing to fully understand their conditioning or intermediate generated scene content. Here, we propose to integrate pre-trained semantic feature extractors to incorporate stronger scene semantics as conditioning to achieve high-quality generation even at distant viewpoints. We investigate two different strategies, (1) warped semantic features and (2) an alternating scheme of understanding and generation at each denoising step. Experimental results on multiple datasets demonstrate the clear qualitative and quantitative (4.69%-15.26% in FID) improvement over state-of-the-art alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。