用语义提示增强稀疏输入下的3D高斯点渲染效果
SPC-GS: Gaussian Splatting with Semantic-Prompt Consistency for Indoor Open-World Free-view Synthesis from Sparse Inputs
- 基于场景布局初始化与语义提示一致性正则化
- 重建质量提升3.06 dB，语义分割准确率提高7.3%
- 适合稀疏输入的室内开放世界自由视角合成
基于3D高斯点渲染的室内开放世界自由视角合成方法在密集输入下表现优异，但在稀疏输入下效果不佳，主要由于高斯点分布稀疏和视图监督不足。为此，我们提出SPC-GS，结合场景布局初始化（SGI）与语义提示一致性（SPC）正则化，实现稀疏输入下的开放世界自由视角合成。SGI通过视频生成模型生成的视角变换图像和视图约束的高斯点细化，实现基于场景布局的密集高斯分布；SPC利用SAM2构建的语义提示一致性约束，将训练视图中的可用语义作为指导提示，在新视图中优化视觉重叠区域，同时满足2D与3D一致性。大量实验表明，SPC-GS在Replica和ScanNet基准上表现卓越，重建质量提升3.06 dB PSNR，开放世界语义分割的mIoU提升7.3%。
原文摘要 · Abstract (English)
3D Gaussian Splatting-based indoor open-world free-view synthesis approaches have shown significant performance with dense input images. However, they exhibit poor performance when confronted with sparse inputs, primarily due to the sparse distribution of Gaussian points and insufficient view supervision. To relieve these challenges, we propose SPC-GS, leveraging Scene-layout-based Gaussian Initialization (SGI) and Semantic-Prompt Consistency (SPC) Regularization for open-world free view synthesis with sparse inputs. Specifically, SGI provides a dense, scene-layout-based Gaussian distribution by utilizing view-changed images generated from the video generation model and view-constraint Gaussian points densification. Additionally, SPC mitigates limited view supervision by employing semantic-prompt-based consistency constraints developed by SAM2. This approach leverages available semantics from training views, serving as instructive prompts, to optimize visually overlapping regions in novel views with 2D and 3D consistency constraints. Extensive experiments demonstrate the superior performance of SPC-GS across Replica and ScanNet benchmarks. Notably, our SPC-GS achieves a 3.06 dB gain in PSNR for reconstruction quality and a 7.3% improvement in mIoU for open-world semantic segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。