让大模型用数学公式固定相机参数,提升3D定位准确性。
Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs

- 用公式显式写出相机投影关系,把工具返回的深度和内参当变量代入。
- 在相机参数变化0.5倍到1.5倍时,3D检测与定位效果显著优于基线。
- 适合需要稳定3D理解的视觉-语言模型应用,如自动驾驶、机器人导航。
多模态大模型(MLLMs)进行3D定位(包括3D目标检测与3D视觉定位)受限于相机内参模糊性:同一图像在不同相机下可能对应不同3D场景。现有方法或忽略相机参数而过拟合训练时的默认内参,或从外部工具获取深度与3D线索但仅将其作为数值提示(模型可自由解释),导致相机信息无法确定性地传入预测结果。本文提出一种方程锚定的工具使用框架,将空间工具重新视为公式变量。该框架主动获取相机内参并采样多点度量深度,在思维链(CoT)中显式写出针孔逆投影公式 $\hat{X} = (u_c - c_x)\bar{Z}/f_x$,并将工具输出代入公式后回归最终的9-自由度边界框。在相机内参缩放范围为0.5×至1.5×的3D目标检测与3D视觉定位任务上,本方法显著优于仅使用RGB或工具增强的基线,尤其在相机偏离训练尺度最大时表现更优。代码与数据将公开。
原文摘要 · Abstract (English)
3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly), both preventing camera information from being deterministically propagated into the prediction. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation $\hat{X} = (u_c - c_x)\bar{Z}/f_x$ explicitly in Chain-of-Thought (CoT), and substitutes tool outputs into the formula before regressing the final 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from $0.5\times$ to $1.5\times$, our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。