让大模型通过主动探索，从零开始找到目标视角。
Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?
- 设计新任务TVR，让模型主动调整视角直到匹配目标图像
- 现有最强模型成功率仅12.0%，且移动比旋转更难复现
- 自研训练框架提升至51.4%，适合研究具身智能的学者
人类可通过头部和身体运动复现目标图像的视角，但当前基础模型的空间智能多为被动理解预采集观测。本文提出目标视角复现（TVR）任务：代理在3D环境中主动调整视角，直至观察结果与给定目标图像一致，并构建了涵盖场景尺度与目标视图丰富度的室内仿真基准TVRBench。评估显示，最强开源与闭源模型成功率分别仅为7.8%和12.0%。细粒度分析发现两大瓶颈：模型难以处理多轮视觉历史，且当需身体位移而非原地旋转时性能急剧下降，暴露出空间差异到具身动作映射的差距。为此，我们构建统一的TVR后训练框架，包含专家轨迹SFT、理由监督的CoT-SFT、离线单轮GRPO及在线多轮GRPO。视觉-动作SFT带来主要提升，使9B开源模型成功率升至50.8%；多轮GRPO实现跨房间精细化优化，总体达51.4%；而CoT监督与单轮GRPO反而降低闭环表现。结果确立了TVRBench作为衡量和训练具身感知与行动能力的基础模型的测试平台。代码、数据与模型已公开于https://github.com/aim-uofa/TVRBench。
原文摘要 · Abstract (English)
Humans can reproduce the viewpoint specified by a target image through active head and body motion, yet spatial intelligence in foundation models has largely been studied as passive understanding of pre-collected observations. We introduce Target Viewpoint Reproduction (TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- and TVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turn visual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVR post-training framework covering expert-trajectory SFT, rationale-supervised CoT-SFT, offline Single-turn GRPO, and on-policy Multi-turn GRPO from live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establish TVRBench as a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。