将图像特征高效映射到3D点云,解决多视角不一致问题。
Splat Feature Solver
- 把特征提升建模为稀疏线性逆问题,闭式求解效率高。
- 在开放词汇3D分割任务上超越现有方法,结果稳定且快速生成。
- 适合需要高质量3D特征的视觉理解研究者使用。
特征提升已成为3D场景理解的关键环节,能将丰富的图像特征描述符(如DINO、CLIP)附加到基于点的3D表示中。核心挑战在于如何在多视角图像不一致的情况下,最优地为3D原型分配通用属性。本文提出一种统一的、核函数与特征无关的特征提升形式化方法,将其建模为稀疏线性逆问题,并可在闭式下高效求解。该方法在凸损失下具有全局最优误差的可证明上界,确保提升特征质量。为缓解多视角观测中的不一致性与噪声,引入两种互补正则化策略:Tikhonov Guidance通过软对角占优增强数值稳定性;Post-Lifting Aggregation利用特征聚类过滤噪声输入。大量实验证明,本方法在开放词汇3D分割基准上达到当前最优性能,显著优于基于训练、分组和启发式前向的基线方法,且特征生成仅需数分钟。代码与可视化资源已公开于GitHub及官网。
原文摘要 · Abstract (English)
Feature lifting has emerged as a crucial component in 3D scene understanding, enabling the attachment of rich image feature descriptors (e.g., DINO, CLIP) onto splat-based 3D representations. The core challenge lies in optimally assigning rich general attributes to 3D primitives while addressing the inconsistency issues from multi-view images. We present a unified, kernel- and feature-agnostic formulation of the feature lifting problem as a sparse linear inverse problem, which can be solved efficiently in closed form. Our approach admits a provable upper bound on the global optimal error under convex losses for delivering high quality lifted features. To address inconsistencies and noise in multi-view observations, we introduce two complementary regularization strategies to stabilize the solution and enhance semantic fidelity. Tikhonov Guidance enforces numerical stability through soft diagonal dominance, while Post-Lifting Aggregation filters noisy inputs via feature clustering. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on open-vocabulary 3D segmentation benchmarks, outperforming training-based, grouping-based, and heuristic-forward baselines while producing lifted features in minutes. Our \textbf{code} is available in the \href{https://github.com/saliteta/splat-distiller/tree/main}{\textcolor{blue}{GitHub}}. We provide additional \href{https://splat-distiller.pages.dev/}{\textcolor{blue}{website}} for more visualization, as well as the \href{https://www.youtube.com/watch?v=CH-G5hbvArM}{\textcolor{blue}{video}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。