arXiv:2606.03100cs.CVcs.LG2026-06中稿 · ICML

通过分层筛选视图与关键令牌,提升零样本3D问答的细节保留能力。

Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation

论文配图:Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation
图 1 · 摘自论文原文
  • 分层筛选:结合语义与几何信息选择重要视图
  • 用最优传输框架选出跨视图代表性令牌,减少冗余
  • 无需微调,在多个基准上接近训练方法性能

近期,利用2D视觉语言模型(VLMs)进行零样本3D场景理解受到广泛关注,因其出色的空间推理能力。通常从3D点云中采样多个2D视图,输入预训练的VLMs以回答问题。该范式强调输入上下文质量的重要性,同时面临在有限输入预算下保留尽可能多任务相关3D细节的挑战。我们提出 exttt{KeyVT},一种在视图和令牌两个层级进行输入上下文收集的分层方法。具体而言,将像素特征与相机参数结合,基于语义内容和几何位置评估视图重要性,生成空间一致且任务相关的视图。此外,我们通过最优传输(OT)框架解决所选视图间补丁的冗余问题,将视图令牌与关键令牌建模为嵌入空间中的两个离散分布,以最小化OT距离来期望覆盖所有视图特征。我们在三个广泛使用的基准上评估了该框架,结果表明其显著优于现有无微调方法,并达到与基于训练的方法相当的性能。

原文摘要 · Abstract (English)

Recently, zero-shot 3D scene understanding via 2D Vision-Language Models (VLMs) has gained increasing research interest due to their promising spatial reasoning capabilities. Typically, multiple 2D views are sampled from a 3D point cloud and fed into pre-trained VLMs to answer a given question. This paradigm highlights the critical role of input context quality and raises the challenge of retaining as many task-relevant 3D details as possible under a limited input budget. We propose \texttt{KeyVT}, a hierarchical approach for input context collection at both the view and token levels. Specifically, we combine pixel features with camera parameters and assess view importance based on both semantic content and geometric position, resulting in spatially consistent and task-relevant views. Furthermore, we address redundancy among patches across selected views by identifying representative tokens under the optimal transport (OT) framework, where view tokens and key tokens are formulated as two discrete distributions in the embedding space. These key tokens are expected to cover all view features by minimizing the OT distance. We evaluate our framework on three widely used benchmarks, demonstrating significant improvements over existing tuning-free methods and performance comparable to training-based approaches.

3D问答视觉语言模型最优传输零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。