用轻量适配器实现稀疏视角3D高斯点渲染,节省显存还画得更好
MuSASplat: Efficient Sparse-View 3D Gaussian Splats via Lightweight Multi-Scale Adaptation
- 设计轻量多尺度适配器,仅需少量参数微调大模型
- 在极稀疏输入下仍保持高质量新视角生成,显存占用降低超60%
- 适合资源有限但需高保真3D重建的科研与工业应用
稀疏视角3D高斯点渲染旨在从有限输入图像中生成高质量新视角。尽管近期无需位姿的前馈方法借助预训练3D先验已取得显著成果,但多数方法依赖大型Vision Transformer(ViT)主干网络的完整微调,带来高昂的GPU开销。本文提出MuSASplat框架,大幅降低无位姿前馈3D高斯点模型的训练计算负担,同时保持良好渲染质量。核心是轻量级多尺度适配器,可高效微调基于ViT的架构,仅需极少训练参数。该设计避免了以往全模型适配带来的巨大显存开销,即便在极稀疏输入下也能保持高保真新视角合成。此外,引入特征融合聚合器,有效整合多视角特征,相比广泛使用的记忆库,其几何一致性更强,显著降低内存占用、训练复杂度和计算成本。在多个数据集上的大量实验表明,MuSASplat达到当前最优渲染质量,但参数量和训练资源需求远低于现有方法。
原文摘要 · Abstract (English)
Sparse-view 3D Gaussian splatting seeks to render high-quality novel views of 3D scenes from a limited set of input images. While recent pose-free feed-forward methods leveraging pre-trained 3D priors have achieved impressive results, most of them rely on full fine-tuning of large Vision Transformer (ViT) backbones and incur substantial GPU costs. In this work, we introduce MuSASplat, a novel framework that dramatically reduces the computational burden of training pose-free feed-forward 3D Gaussian splats models with little compromise of rendering quality. Central to our approach is a lightweight Multi-Scale Adapter that enables efficient fine-tuning of ViT-based architectures with only a small fraction of training parameters. This design avoids the prohibitive GPU overhead associated with previous full-model adaptation techniques while maintaining high fidelity in novel view synthesis, even with very sparse input views. In addition, we introduce a Feature Fusion Aggregator that integrates features across input views effectively and efficiently. Unlike widely adopted memory banks, the Feature Fusion Aggregator ensures consistent geometric integration across input views and meanwhile mitigates the memory usage, training complexity, and computational costs significantly. Extensive experiments across diverse datasets show that MuSASplat achieves state-of-the-art rendering quality but has significantly reduced parameters and training resource requirements as compared with existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。