用稀疏图像实现通用3D场景语义重建,无需针对每场景优化。
SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understanding
- 双分支结构共享浅层卷积,融合颜色与语义特征。
- 在多个数据集上达到领先性能,推理速度快且泛化能力强。
- 适合需要快速部署的机器人视觉系统使用。
3D场景的语义理解对机器人在复杂环境中安全高效运行至关重要。现有语义场景重建与语义感知的新视角合成方法通常依赖密集多视角输入,并需针对每个场景进行优化,限制了其在真实应用中的实用性与可扩展性。为此,我们提出SemGS,一种从稀疏图像输入中重建通用语义场的前馈框架。SemGS采用双分支架构提取颜色与语义特征,两分支共享浅层CNN层,使语义推理能够利用颜色外观中的纹理与结构线索。我们还引入相机感知注意力机制到特征提取器中,显式建模相机视点间的几何关系。提取的特征解码为共享几何一致性的双高斯,同时保留分支特有属性,并进一步光栅化生成新视角下的语义图。此外,引入区域平滑损失以增强语义一致性。实验表明,SemGS在基准数据集上达到最先进性能,同时具备快速推理能力及在多样合成与真实场景中的强泛化能力。
原文摘要 · Abstract (English)
Semantic understanding of 3D scenes is essential for robots to operate effectively and safely in complex environments. Existing methods for semantic scene reconstruction and semantic-aware novel view synthesis often rely on dense multi-view inputs and require scene-specific optimization, limiting their practicality and scalability in real-world applications. To address these challenges, we propose SemGS, a feed-forward framework for reconstructing generalizable semantic fields from sparse image inputs. SemGS uses a dual-branch architecture to extract color and semantic features, where the two branches share shallow CNN layers, allowing semantic reasoning to leverage textural and structural cues in color appearance. We also incorporate a camera-aware attention mechanism into the feature extractor to explicitly model geometric relationships between camera viewpoints. The extracted features are decoded into dual-Gaussians that share geometric consistency while preserving branch-specific attributes, and further rasterized to synthesize semantic maps under novel viewpoints. Additionally, we introduce a regional smoothness loss to enhance semantic coherence. Experiments show that SemGS achieves state-of-the-art performance on benchmark datasets, while providing rapid inference and strong generalization capabilities across diverse synthetic and real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。