用单目深度模型提升3D高斯点云的泛化能力,无需场景优化。
MonoSplat: Generalizable 3D Gaussian Splatting from Monocular Depth Foundation Models
- 利用预训练单目深度模型提取视觉先验,增强特征表达。
- 轻量注意力机制实现跨视角特征对齐,生成更精确的高斯点。
- 在多个真实数据集上表现优异,适合实时高保真渲染应用。
近期可泛化的3D高斯点云渲染方法在无需场景优化的情况下实现了实时高保真渲染,但现有方法在新场景推理时仍难以处理陌生视觉内容,泛化能力有限。为此,我们提出MonoSplat,一种利用预训练单目深度基础模型中丰富视觉先验的鲁棒高斯重建框架。其包含两个关键组件:将单目特征转换为多视图表示的Mono-Multi Feature Adapter,以及融合两类特征以精准生成高斯点的Integrated Gaussian Prediction模块。通过Adapter的轻量级注意力机制,特征在视间实现无缝对齐与聚合,同时保留重要单目先验,使Prediction模块能够生成几何与外观更准确的高斯基元。在多个真实世界数据集上的大量实验表明,MonoSplat在重建质量与泛化能力方面均优于现有方法,且仅需极少可训练参数,计算效率高。代码已公开于https://github.com/CUHK-AIM-Group/MonoSplat。
原文摘要 · Abstract (English)
Recent advances in generalizable 3D Gaussian Splatting have demonstrated promising results in real-time high-fidelity rendering without per-scene optimization, yet existing approaches still struggle to handle unfamiliar visual content during inference on novel scenes due to limited generalizability. To address this challenge, we introduce MonoSplat, a novel framework that leverages rich visual priors from pre-trained monocular depth foundation models for robust Gaussian reconstruction. Our approach consists of two key components: a Mono-Multi Feature Adapter that transforms monocular features into multi-view representations, coupled with an Integrated Gaussian Prediction module that effectively fuses both feature types for precise Gaussian generation. Through the Adapter's lightweight attention mechanism, features are seamlessly aligned and aggregated across views while preserving valuable monocular priors, enabling the Prediction module to generate Gaussian primitives with accurate geometry and appearance. Through extensive experiments on diverse real-world datasets, we convincingly demonstrate that MonoSplat achieves superior reconstruction quality and generalization capability compared to existing methods while maintaining computational efficiency with minimal trainable parameters. Codes are available at https://github.com/CUHK-AIM-Group/MonoSplat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。