让AI从语言描述精准推断相机位姿,实现多视角空间推理
Predicting Camera Pose from Perspective Descriptions for Spatial Reasoning
- 用相机位姿显式锚定多视图信息,实现跨视角融合
- 语言描述转位姿精度超90%,推理速度提升177倍
- 适合需要快速交互的智能导航与机器人视觉任务
多图像空间推理仍是当前多模态大模型的难点。尽管单视图感知本质为2D,但多视图推理需构建跨视角的一致场景理解。本文研究视角转换问题:模型需从多视图观测中建立一致的3D理解,并基于语言指定的新视角进行推理。提出CAMCUE框架,通过显式注入相机位姿作为几何锚点,将视觉特征与位姿对齐,将自然语言视角描述映射到目标相机位姿,并生成条件化的想象视图以支持回答。为此,构建了包含27,668个训练样本和508个测试样本的CAMCUE-DATA数据集,涵盖多视图图像、位姿及多样化的语言视角描述与视角变换问题。测试集还包含人工标注的视角描述,用于评估对人类语言的泛化能力。CAMCUE整体准确率提升9.06%,语言描述转位姿的旋转误差在20°内超过90%准确率,平移误差在0.5阈值内表现优异。该方法避免了昂贵的测试时搜索匹配,使单例推理时间从256.6秒降至1.45秒,支持实时交互应用。
原文摘要 · Abstract (English)
Multi-image spatial reasoning remains challenging for current multimodal large language models (MLLMs). While single-view perception is inherently 2D, reasoning over multiple views requires building a coherent scene understanding across viewpoints. In particular, we study perspective taking, where a model must build a coherent 3D understanding from multi-view observations and use it to reason from a new, language-specified viewpoint. We introduce CAMCUE, a pose-aware multi-image framework that uses camera pose as an explicit geometric anchor for cross-view fusion and novel-view reasoning. CAMCUE injects per-view pose into visual tokens, grounds natural-language viewpoint descriptions to a target camera pose, and synthesizes a pose-conditioned imagined target view to support answering. To support this setting, we curate CAMCUE-DATA with 27,668 training and 508 test instances pairing multi-view images and poses with diverse target-viewpoint descriptions and perspective-shift questions. We also include human-annotated viewpoint descriptions in the test split to evaluate generalization to human language. CAMCUE improves overall accuracy by 9.06% and predicts target poses from natural-language viewpoint descriptions with over 90% rotation accuracy within 20° and translation accuracy within a 0.5 error threshold. This direct grounding avoids expensive test-time search-and-match, reducing inference time from 256.6s to 1.45s per example and enabling fast, interactive use in real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。