测试大模型在全景图像中的视角依赖空间推理能力，发现其表现严重受限。
Beyond Localization: A Comprehensive Diagnosis of Perspective-Conditioned Spatial Reasoning in MLLMs from Omnidirectional Images
- 构建全景图像诊断基准PCSR-Bench，包含8万多问题对
- 模型在方向推理任务中准确率低至0.64%，存在显著感知-推理差距
- 强化学习可部分提升性能，但效果受任务和奖励设计影响
多模态大模型虽具强大视觉感知能力，但在不同视角下的空间推理仍显不足。本文以360度全景图像为场景，研究视角依赖的空间推理（PCSR）问题，因其覆盖范围广，能减少局部观察的模糊性，但仍需依赖视角判断。为此，我们构建了PCSR-Bench，涵盖2,600张全景图、26个室内环境，共84,373个问答对，分为八个任务，归于感知、空间与高级PCSR三类认知组。评估14个代表性MLLM后发现：在有限视场推理（T7）上准确率达57.59%，但在相对方向（T2）、自我中心旋转（T4）及开放组合方向链（T3）任务中分别降至13.49%、7.13%和0.64%。进一步对7B规模模型进行基于强化学习的诊断，通过奖励重塑使基准模型从31.10%提升至60.06%，表明PCSR具备一定可塑性，但改善效果具有任务特异性，对奖励设计和评估协议敏感。结果表明PCSR是当前多模态模型的关键瓶颈，优化潜力存在但有限。数据与代码详见https://github.com/Caleb-ychen/PCSR-Benchmark。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) show strong visual perception, yet remain limited in reasoning about space under changing viewpoints. We study this challenge as Perspective-Conditioned Spatial Reasoning (PCSR) in 360 degree omnidirectional images, where broad scene coverage reduces ambiguity from partial observations without eliminating the need for viewpoint-dependent inference. To assess this capability, we introduce PCSR-Bench, a diagnostic benchmark of 84,373 QA pairs from 2,600 omnidirectional images across 26 indoor environments, organized into eight tasks under three cognitive groups--- Perception, Spatial, and advanced PCSR. We evaluate 14 representative MLLMs and observe a substantial perception--reasoning gap: accuracy reaches 57.59% on Limited Field-of-View Reasoning (T7) but drops to 13.49%, 7.13%, and 0.64% on Relative Direction (T2), Egocentric Rotation (T4), and open-ended Compositional Directional Chains (T3), respectively. To probe the plasticity of this gap, we conduct an RL-based diagnostic study on a 7B-scale model. Reward shaping improves a matched 7B baseline from 31.10% to 60.06% under a controlled setting, suggesting that PCSR exhibits partial plasticity rather than being fully immutable. Still, these gains are task-selective, sensitive to reward design, and partially dependent on the evaluation protocol. These results position PCSR as a key bottleneck in current MLLMs and highlight meaningful yet bounded room for recovery under targeted optimization. Details and access are available at https://github.com/Caleb-ychen/PCSR-Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。