用预训练通用编码器替代场景特有编码器,实现无需重训的3D语言场快速构建。
Gen-LangSplat: Generalized Language Gaussian Splatting with Pre-Trained Feature Compression
- 采用预训练通用自编码器替代逐场景训练的编码器,统一压缩语言特征。
- 在新场景中无需微调,仍能保持与原方法相当甚至更优的查询性能。
- 适合需快速部署、支持多场景交互的实时3D AI应用开发者。
在3D环境中建模开放词汇语言场对实现直观的人机交互与环境查询至关重要。当前先进方法如LangSplat利用3D高斯点云高效构建语言场,通过CLIP等高维模型提取特征进行编码。然而,该方法依赖为每个场景单独训练语言自编码器进行特征压缩,带来高昂的逐场景优化开销,严重制约部署可扩展性。本文提出Gen-LangSplat,通过将场景特定自编码器替换为在大规模ScanNet数据集上预训练的通用自编码器,实现跨场景固定紧凑的潜在空间表示,无需任何场景定制训练。这一架构变革使整个语言场构建流程效率显著提升,同时在查询性能上达到或超越原始LangSplat。我们通过详尽消融实验确定最优潜在嵌入维度,并使用均方误差和原始512维CLIP嵌入与重构嵌入间的余弦相似度评估表征保真度。结果表明,通用嵌入能高效且准确地支持新3D场景中的开放词汇查询,为可扩展、实时交互式3D AI应用铺平道路。
原文摘要 · Abstract (English)
Modeling open-vocabulary language fields in 3D is essential for intuitive human-AI interaction and querying within physical environments. State-of-the-art approaches, such as LangSplat, leverage 3D Gaussian Splatting to efficiently construct these language fields, encoding features distilled from high-dimensional models like CLIP. However, this efficiency is currently offset by the requirement to train a scene-specific language autoencoder for feature compression, introducing a costly, per-scene optimization bottleneck that hinders deployment scalability. In this work, we introduce Gen-LangSplat, that eliminates this requirement by replacing the scene-wise autoencoder with a generalized autoencoder, pre-trained extensively on the large-scale ScanNet dataset. This architectural shift enables the use of a fixed, compact latent space for language features across any new scene without any scene-specific training. By removing this dependency, our entire language field construction process achieves a efficiency boost while delivering querying performance comparable to, or exceeding, the original LangSplat method. To validate our design choice, we perform a thorough ablation study empirically determining the optimal latent embedding dimension and quantifying representational fidelity using Mean Squared Error and cosine similarity between the original and reprojected 512-dimensional CLIP embeddings. Our results demonstrate that generalized embeddings can efficiently and accurately support open-vocabulary querying in novel 3D scenes, paving the way for scalable, real-time interactive 3D AI applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。