无需训练即可生成镜面场景中反射一致的新视角。
Reflection-aware Generative Novel View Synthesis
- 将镜像视为互补视图，构建虚拟视角对齐。
- 提出双阶段生成方法，实现反射内容一致的图像生成。
- 适用于真实与合成镜面场景，无需微调模型。
我们提出 Ref-GeNVS，一种无需训练的、能感知镜面的生成式新视角合成方法，用于镜面场景。现有多视角扩散模型常无法识别场景中的镜子，也无法利用反射内容进行场景生成。为解决此问题且不需额外训练，我们的核心思想是将镜像视图视为两个互补视角。从输入图像中估计镜面平面，并反射相机位姿以构建虚拟视角。基于此虚拟视角设定，我们提出两阶段生成方法：镜面门控注意力与反射注入，通过显式利用多视角扩散模型中的反射关系，实现反射一致的新视角合成。Ref-GeNVS继承了多视角扩散模型的强大泛化能力，且无需微调。在包含镜子的合成与真实场景中，其性能优于近期生成式新视角合成方法，能够生成反射一致且上下文连贯的新视角，揭示仅通过镜子可见的场景结构。
原文摘要 · Abstract (English)
We propose Ref-GeNVS, a training-free, reflection-aware method for generative novel view synthesis (NVS) in mirror scenes. Existing multi-view diffusion models often fail to recognize the mirror in the scene and cannot exploit reflected content for scene generation. To fix this issue without additional training, our key idea is to treat a mirror image as two complementary views. From input images, we estimate the mirror plane and reflect camera poses to form virtual views. Based on this virtual view setup, we propose a two-stage generation method consisting of Mirror-gated attention and Reflection injection, which enables reflection-consistent NVS by explicitly leveraging reflection relationships in a multi-view diffusion model. Ref-GeNVS inherits the strong generalizability of the multi-view diffusion backbone, while it does not require finetuning. On synthetic and real scenes including mirrors, Ref-GeNVS outperforms recent generative NVS methods by generating reflection-consistent and contextually coherent novel views, revealing scene structure visible only through mirrors. Project page: https://kim-geonu.github.io/Ref-GeNVS/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。