让虚拟现实中的户外场景可编辑,通过语义分割分离大块场景。
Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality

- 基于语义控制的高斯点云分割,支持自然户外场景的大块分离。
- 在自建数据集上视觉质量超越当前最佳,在3D-OVS上分割精度领先。
- 用户实测显示自由移动时更偏好该方法,体验更佳。
高斯点云(Gaussian Splatting, GS)技术推动了虚拟现实中的新视角合成与实时渲染。然而,由GS生成的3D环境难以编辑。为实现场景增强或添加3D资产,按类别分割高斯点是关键。现有分割方法通常局限于特定场景(如“环形”场景),难以处理大型户外非环形场景中大物体的移除。本文提出语义控制的高斯点云(SCGS),一种面向无约束自然环境的分割驱动型方法,可实现大场景部分的分离。该方法支持场景编辑与场景部件提取,适用于虚拟现实应用。同时,我们构建了一个具有挑战性的户外数据集,突破传统“环形”设置限制。在自建数据集上,我们的方法在视觉质量上优于现有最佳;在3D-OVS数据集上,分割质量也表现领先。我们开展了探索性用户研究,对比固定视角下360视频、普通GS与SCGS的效果;随后进行主实验,允许用户自由移动,评估普通GS与SCGS。结果表明,参与者显著偏好SCGS。整体而言,该方法在技术和用户体验上均超越当前最佳水平。
原文摘要 · Abstract (English)
Advancements in 3D rendering like Gaussian Splatting (GS) allow novel view synthesis and real-time rendering in virtual reality (VR). However, GS-created 3D environments are often difficult to edit. For scene enhancement or to incorporate 3D assets, segmenting Gaussians by class is essential. Existing segmentation approaches are typically limited to certain types of scenes, e.g., ''circular'' scenes, to determine clear object boundaries. However, this method is ineffective when removing large objects in non-''circling'' scenes such as large outdoor scenes. We propose Semantics-Controlled GS (SCGS), a segmentation-driven GS approach, enabling the separation of large scene parts in uncontrolled, natural environments. SCGS allows scene editing and the extraction of scene parts for VR. Additionally, we introduce a challenging outdoor dataset, overcoming the ''circling'' setup. We outperform the state-of-the-art in visual quality on our dataset and in segmentation quality on the 3D-OVS dataset. We conducted an exploratory user study, comparing a 360-video, plain GS, and SCGS in VR with a fixed viewpoint. In our subsequent main study, users were allowed to move freely, evaluating plain GS and SCGS. Our main study results show that participants clearly prefer SCGS over plain GS. We overall present an innovative approach that surpasses the state-of-the-art both technically and in user experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。