用视图间几何关系提升单图新视角生成的一致性,无需重训练。
Synthesizing Consistent Novel Views via 3D Epipolar Attention without Re-Training
- 通过视极几何定位参考视图重叠区域信息,注入目标视图生成
- 在未微调情况下显著提升新视角一致性,下游3D重建性能也增强
- 适用于希望快速部署高一致性生成模型的研究者与开发者
大规模扩散模型在单图新视角合成中展现出出色的零样本能力。然而,这些模型常因未能充分利用参考视图的上下文信息而导致新旧视图间不一致。当两视图视野有重叠时,需保证对应区域在几何与外观上一致。为此,本文提出基于视极几何的方法,定位并检索输入视图中的重叠信息,并将其融入目标视图生成过程,整个流程无学习参数,无需训练或微调。进一步,将该机制扩展至多视图设置,可从输入视图及其他目标视图中获取重叠信息,增强整体一致性。定性与定量实验表明,该方法在不进行任何微调的情况下显著提升了合成视图的一致性,并改善了3D重建等下游任务表现。代码已公开于https://github.com/botaoye/ConsisSyn。
原文摘要 · Abstract (English)
Large diffusion models demonstrate remarkable zero-shot capabilities in novel view synthesis from a single image. However, these models often face challenges in maintaining consistency across novel and reference views. A crucial factor leading to this issue is the limited utilization of contextual information from reference views. Specifically, when there is an overlap in the viewing frustum between two views, it is essential to ensure that the corresponding regions maintain consistency in both geometry and appearance. This observation leads to a simple yet effective approach, where we propose to use epipolar geometry to locate and retrieve overlapping information from the input view. This information is then incorporated into the generation of target views, eliminating the need for training or fine-tuning, as the process requires no learnable parameters. Furthermore, to enhance the overall consistency of generated views, we extend the utilization of epipolar attention to a multi-view setting, allowing retrieval of overlapping information from the input view and other target views. Qualitative and quantitative experimental results demonstrate the effectiveness of our method in significantly improving the consistency of synthesized views without the need for any fine-tuning. Moreover, This enhancement also boosts the performance of downstream applications such as 3D reconstruction. The code is available at https://github.com/botaoye/ConsisSyn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。