通过测试时优化噪声，提升单图像新视角生成的几何一致性。
Geometric Consistency Refinement for Single Image Novel View Synthesis via Test-Time Adaptation of Diffusion Models
- 在采样过程中优化初始噪声以满足视点间的极线约束。
- 在MegaScenes数据集上显著改善几何一致性，且保持图像质量。
- 无需训练或微调模型，适用于多种主流NVS扩散模型。
用于单图像新视角合成（NVS）的扩散模型虽能生成高度逼真的图像，但在相对姿态的几何一致性方面存在局限。生成图像常违反由目标姿态决定的极线约束。本文提出一种方法，通过基于图像匹配和极线约束的损失函数，在扩散采样过程中优化初始噪声，使生成图像既真实又符合给定目标姿态的几何约束。该方法无需训练数据或模型微调，可应用于多种前沿的单图像NVS扩散模型。在MegaScenes数据集上的实验表明，相比基线模型，几何一致性显著提升，同时保持了高质量的生成效果。
原文摘要 · Abstract (English)
Diffusion models for single image novel view synthesis (NVS) can generate highly realistic and plausible images, but they are limited in the geometric consistency to the given relative poses. The generated images often show significant errors with respect to the epipolar constraints that should be fulfilled, as given by the target pose. In this paper we address this issue by proposing a methodology to improve the geometric correctness of images generated by a diffusion model for single image NVS. We formulate a loss function based on image matching and epipolar constraints, and optimize the starting noise in a diffusion sampling process such that the generated image should both be a realistic image and fulfill geometric constraints derived from the given target pose. Our method does not require training data or fine-tuning of the diffusion models, and we show that we can apply it to multiple state-of-the-art models for single image NVS. The method is evaluated on the MegaScenes dataset and we show that geometric consistency is improved compared to the baseline models while retaining the quality of the generated images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。