arXiv:2608.00110cs.CV2026-08

用3D训练数据提升纯RGB模型的空间理解能力,无需推理时额外输入。

Distill What RGB Can Recover: Privileged 3D Evidence for RGB-Only Vision-Language Models

论文配图:Distill What RGB Can Recover: Privileged 3D Evidence for RGB-Only Vision-Language Models
图 1 · 摘自论文原文
  • 通过蒸馏3D证据构建可部署的纯RGB模型,保留空间推理能力。
  • 在4个基准上,纯RGB学生模型全胜基线,最高提升19.1分。
  • 适合希望低成本部署空间感知视觉语言模型的研究者。

3D场景理解需推理实体存在、空间布局与物体关系,但仅凭RGB图像常缺乏足够3D线索。现有3D-VLM通常在推理时依赖深度或3D位置信息,引入额外采集、重建或标注成本,限制纯RGB部署。本文研究如何将训练时的3D证据转化为推理时仅用RGB的时空推理能力。提出特权证据蒸馏框架,通过统一证据接口与受控残差注入构建可蒸馏教师模型,并将其知识通过logit与结构化表示蒸馏转移至仅接收RGB图像和问题的学生模型。为避免模仿RGB无法支持的教师信号,进一步引入基于证据敏感性的蒸馏,利用损坏的证据识别高度依赖证据的目标并降低其监督权重。还基于匹配基线、教师与学生定义可恢复性分解,将特权增益分为RGB可恢复改进与残差教师优势。在4个基准上,教师在11项指标中7项最优。纯RGB学生模型在全部11项指标上优于匹配基线,包括10.4(ScanQA CIDEr)与19.1(Scan2Cap [email protected])的提升,无需额外推理输入。结果验证了训练时3D证据蒸馏的有效性。此外,基线-教师-学生分析刻画了不同证据类型与空间技能的增益传递路径。

原文摘要 · Abstract (English)

3D scene understanding requires reasoning about entity existence, spatial layout, and object relations, yet RGB images alone often provide insufficient 3D cues. Existing 3D-VLMs commonly rely on depth or 3D-position-aware inputs at inference time, introducing additional acquisition, reconstruction, or annotation costs that limit RGB-only deployment. We therefore study how training-time 3D evidence can be converted into spatial reasoning capabilities retained under RGB-only inference. We propose a privileged-evidence distillation framework that constructs a distillable teacher through a unified evidence interface and controlled residual injection, and transfers its knowledge to a deployable student receiving only RGB images and questions through logit and structured representation distillation. To avoid imitating teacher signals unsupported by RGB, we further introduce evidence-sensitivity-guided distillation, which uses corrupted evidence to identify highly evidence-dependent targets and down-weight their supervision. We also define a recoverability decomposition based on the matched baseline, teacher, and student, separating privileged gains into RGB-recoverable improvements and residual teacher advantages. Across four benchmarks, the teacher achieves the best result on 7 of 11 reported metrics among the compared methods. The RGB-only student outperforms its matched baseline on all 11 metrics, including gains of 10.4 ScanQA CIDEr and 19.1 Scan2Cap [email protected], without additional inference-time inputs. These results validate the effectiveness of training-time privileged 3D evidence distillation for both teacher performance and deployable RGB-only spatial reasoning. Separately, our matched baseline-teacher-student analysis characterizes privileged-gain transfer across evidence types and spatial skills.

视觉语言模型3D理解蒸馏纯RGB

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。