用流模型从稀疏重建生成高质量稠密视图,提升3D场景还原效果。
FlowR: Flowing from Sparse to Dense 3D Reconstructions
- 设计多视角流匹配模型,直接映射稀疏到稠密视图渲染结果。
- 在360万图像对上训练,单次前向传播可处理45个960x540视图。
- 适用于虚拟现实等需高保真3D重建的场景,显著减少密集采集需求。
3D高斯点阵可在实时帧率下实现高质量的新视角合成(NVS),但其质量随偏离训练视角而急剧下降。因此,应用如虚拟现实(VR)需要稠密采集以满足高质量预期,而稠密采集成本高昂且耗时。现有方法尝试通过2D生成模型蒸馏或生成额外训练视图来缓解该问题,但这些模型通常仅基于少量参考视图进行噪声到数据的生成,导致幻觉、生成不一致及后续重建伪影。为此,我们提出一种多视角流匹配模型,学习从可能稀疏重建的视图渲染结果到期望稠密重建渲染结果之间的直接映射。该方法可生成一致的补充视图,从而提升重建质量。模型在包含360万图像对的新数据集上训练,可在单张H100 GPU上以540x960分辨率一次性处理最多45个视图(共91K token)。整个流程在稀疏与稠密视图场景下均能持续改善NVS表现,优于多个主流基准上的先前方法。
原文摘要 · Abstract (English)
3D Gaussian splatting enables high-quality novel view synthesis (NVS) at real-time frame rates. However, its quality drops sharply as we depart from the training views. Thus, dense captures are needed to match the high-quality expectations of applications like Virtual Reality (VR). However, such dense captures are very laborious and expensive to obtain. Existing works have explored using 2D generative models to alleviate this requirement by distillation or generating additional training views. These models typically rely on a noise-to-data generative process conditioned only on a handful of reference input views, leading to hallucinations, inconsistent generation results, and subsequent reconstruction artifacts. Instead, we propose a multi-view, flow matching model that learns a flow to directly connect novel view renderings from possibly sparse reconstructions to renderings that we expect from dense reconstructions. This enables augmenting scene captures with consistent, generated views to improve reconstruction quality. Our model is trained on a novel dataset of 3.6M image pairs and can process up to 45 views at 540x960 resolution (91K tokens) on one H100 GPU in a single forward pass. Our pipeline consistently improves NVS in sparse- and dense-view scenarios, leading to higher-quality reconstructions than prior works across multiple, widely-used NVS benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。