用3D重建增强教师模型，让2D视觉模型学会空间感知。
Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation
- 将2D特征直接转为3D高斯表示，快速生成新视角特征图。
- 在深度估计等任务上超越已有方法，提升3D感知与语义表达能力。
- 适合需要3D理解的2D视觉任务，如自动驾驶、机器人视觉。
视觉基础模型（VFMs）在多种下游2D任务中表现优异，但普遍缺乏3D感知能力。为此，我们提出Splat and Distill框架，通过引入快速前馈式3D重建管道，将3D感知能力注入2D VFMs。给定教师模型输出的2D特征，本方法首先将其前馈式地提升为显式的3D高斯表示，再将这些3D特征“投射”到新视角，生成用于监督学生模型的新2D特征图，实现基于几何约束的知识蒸馏。相比以往依赖逐场景优化的慢速方法，该框架避免了特征平均化缺陷，形成教师与学生同步改进的动态学习过程。我们在单目深度估计、表面法向估计、多视图匹配和语义分割等任务上进行综合评估，结果表明该方法显著优于现有工作，在提升3D感知能力的同时，也增强了2D特征的语义丰富性。
原文摘要 · Abstract (English)
Vision Foundation Models (VFMs) have achieved remarkable success when applied to various downstream 2D tasks. Despite their effectiveness, they often exhibit a critical lack of 3D awareness. To this end, we introduce Splat and Distill, a framework that instills robust 3D awareness into 2D VFMs by augmenting the teacher model with a fast, feed-forward 3D reconstruction pipeline. Given 2D features produced by a teacher model, our method first lifts these features into an explicit 3D Gaussian representation, in a feedforward manner. These 3D features are then ``splatted" onto novel viewpoints, producing a set of novel 2D feature maps used to supervise the student model, ``distilling" geometrically grounded knowledge. By replacing slow per-scene optimization of prior work with our feed-forward lifting approach, our framework avoids feature-averaging artifacts, creating a dynamic learning process where the teacher's consistency improves alongside that of the student. We conduct a comprehensive evaluation on a suite of downstream tasks, including monocular depth estimation, surface normal estimation, multi-view correspondence, and semantic segmentation. Our method significantly outperforms prior works, not only achieving substantial gains in 3D awareness but also enhancing the underlying semantic richness of 2D features. Project page is available at https://davidshavin4.github.io/Splat-and-Distill/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。