用扩散模型生成夜景语义特征,提升3D高斯泼溅渲染质量
GlowGS: Generative Semantic Feature Learning for 3D Gaussian Splatting in Nighttime Glow Scenes

- 通过扩散模型合成未知视角,结合视觉基础模型筛选高质量视图
- 构建语义特征库,使3DGS在无真值情况下优化渲染结果
- 特别适合夜间发光场景,改善模糊与结构缺失问题
现有3D高斯泼溅方法在白天场景中能生成高质量新视角,但在夜间发光区域表现不佳,因缺乏纹理、边缘等结构线索。为此,本文提出GlowGS,利用扩散模型和视觉基础模型(VFM)弥补缺失的结构信息。核心思路包括:1)语义特征生成——通过扩散模型从训练视图合成未知相机位姿的新视角,由VFM评估质量并提取鲁棒特征,构建语义特征库;2)新视角语义学习——对渲染的新视角提取语义特征,搜索特征库中最相似项并最小化距离,隐式施加结构约束。实验表明,该方法显著优于现有方法,在夜间发光场景下生成更语义一致、无伪影的3D视图。
原文摘要 · Abstract (English)
Existing 3DGS methods effectively render high-quality novel views in clear-day scenes. However, they struggle with night scenes, particularly in glow regions, due to the lack of structural features such as textures and edges, which are key cues for splatting-based reconstruction. To address this problem, we leverage a diffusion model and a Vision Foundation Model (VFM) to compensate for missing structural cues. Our method consists of two key novel ideas: semantic feature generation and novel-view semantic learning. First, semantic feature generation produces high-quality semantic features as implicit structural cues for novel views. Specifically, a diffusion model synthesizes novel views with unknown camera poses from training views, while a VFM evaluates their quality. Once high-quality novel views are identified, the VFM extracts robust features to construct the semantic feature bank. Second, novel-view semantic learning enables 3DGS to optimize rendered novel views without requiring ground truth. It achieves this by extracting semantic features from a rendered novel view, searching the feature bank for the most similar features, and minimizing their distance. This process enforces implicit structural constraints, ensuring semantically coherent, artifact-free rendered views. Extensive experiments demonstrate the effectiveness of our GlowGS in generating semantically accurate 3D views, showing significant improvements over existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。