arXiv:2511.20648cs.CV2025-11被引 6

将3D检测转化为语言模型的下一步预测,实现开放词汇的3D物体定位。

LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight

  • 用视觉思维链模拟人类从2D到3D的推理过程
  • 在Omni3D上达38.90 AP_3D,较前人提升13.98
  • 零样本泛化强,适合需要3D感知的智能体

为在真实世界中行动,模型需能识别所见物体并定位其三维位置。当前视觉语言模型(VLM)擅长开放式2D描述与定位,但多物体3D检测仍缺失于主流工具箱。本文提出LocateAnything3D,一种原生适配VLM的3D检测方法,将3D检测建模为下一个词预测问题。核心是简短且显式的视觉思维链(Chain-of-Sight, CoS),模仿人类从图像中推理的路径:先在2D中定位物体,再推断其距离、尺寸与姿态。解码器首先输出2D检测作为视觉思维链,随后按由易到难的课程学习策略预测3D框:跨物体采用由近及远顺序,减少早期歧义并符合自身中心视角的实用性;单个物体内部则按中心-尺寸-旋转分步预测,依据稳定性和可学习性排序。该方法保留开放词汇与视觉提示能力,无需专用头部。在挑战性基准Omni3D上,模型取得38.90 AP_3D,较此前最优结果提升13.98绝对点,即使基线已知2D真值亦然。同时具备零样本泛化能力,表现稳健。通过将3D检测转化为结构化的下一步预测任务,LocateAnything3D为模型实现3D感知提供了实用基础。

原文摘要 · Abstract (English)

To act in the world, a model must name what it sees and know where it is in 3D. Today's vision-language models (VLMs) excel at open-ended 2D description and grounding, yet multi-object 3D detection remains largely missing from the VLM toolbox. We present LocateAnything3D, a VLM-native recipe that casts 3D detection as a next-token prediction problem. The key is a short, explicit Chain-of-Sight (CoS) sequence that mirrors how human reason from images: find an object in 2D, then infer its distance, size, and pose. The decoder first emits 2D detections as a visual chain-of-thought, then predicts 3D boxes under an easy-to-hard curriculum: across objects, a near-to-far order reduces early ambiguity and matches ego-centric utility; within each object, a center-from-camera, dimensions, and rotation factorization ranks information by stability and learnability. This VLM-native interface preserves open-vocabulary and visual-prompting capability without specialized heads. On the challenging Omni3D benchmark, our model achieves state-of-the-art results, with 38.90 AP_3D, surpassing the previous best by +13.98 absolute improvement even when the baseline is given ground-truth 2D boxes. It also generalizes zero-shot to held-out categories with strong robustness. By turning 3D detection into a disciplined next-token problem, LocateAnything3D offers a practical foundation for models to perceive in 3D.

3D检测视觉语言模型开放词汇思维链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。