arXiv:2604.15294cs.AI2026-04ACL被引 1

研究大模型如何仅靠文字理解视角旋转,发现其表现远不如人类。

How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study

论文配图:How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study
图 1 · 摘自论文原文
  • 用文本描述视角变化,测试模型推理最终观察位置的能力。
  • 模型准确率远低于人类,关键在视角与观察的绑定失败。
  • 通过精调关键注意力头,提升性能且不丢失通用能力。

过去一年,空间智能备受关注。以往研究多从视觉空间智能角度出发,依赖视觉输入。然而,在无视觉信息情况下,仅凭语言智能能否赋予模型空间智能,以及模型如何处理纯文本输入的任务仍不清楚。本文聚焦语言智能中的基础能力:视角旋转理解(VRU)。具体地,要求LLMs和VLMs根据多步视角旋转与观察的文本描述,推断最终视角并预测对应观察。实验发现,模型表现显著低于人类(人类达100%准确率),表明当前模型在空间智能上存在巨大差距。通过层间探测与头级因果干预分析,我们发现模型虽在隐藏状态中编码视角信息,却难以将视角位置与对应观察绑定,导致后期产生幻觉。最后,针对因果干预识别的关键注意力头进行选择性微调,结果表明该方法有效提升VRU性能,同时避免对通用能力的灾难性遗忘。数据集与代码将开源。

原文摘要 · Abstract (English)

Over the past year, spatial intelligence has drawn increasing attention. Many prior works study it from the perspective of visual-spatial intelligence, where models have access to visuospatial information from visual inputs. However, in the absence of visual information, whether linguistic intelligence alone is sufficient to endow models with spatial intelligence, and how models perform relevant tasks with text-only inputs still remain unexplored. Therefore, in this paper, we focus on a fundamental and critical capability in spatial intelligence from a linguistic perspective: viewpoint rotation understanding (VRU). Specifically, LLMs and VLMs are asked to infer their final viewpoint and predict the corresponding observation in an environment given textual description of viewpoint rotation and observation over multiple steps. We find that both LLMs and VLMs perform poorly on our proposed dataset while human can easily achieve 100% accuracy, indicating a substantial gap between current model capabilities and the requirements of spatial intelligence. To uncover the underlying mechanisms, we conduct a layer-wise probing analysis and head-wise causal intervention. Our findings reveal that although models encode viewpoint information in the hidden states, they appear to struggle to bind the viewpoint position with corresponding observation, resulting in a hallucination in final layers. Finally, we selectively fine-tune the key attention heads identified by causal intervention to improve VRU performance. Experimental results demonstrate that such selective fine-tuning achieves improved VRU performance while avoiding catastrophic forgetting of generic abilities. Our dataset and code will be released at https://github.com/Young-Zhen/VRU_Interpret .

空间智能视角理解可解释性大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。