给视觉语言模型加空间符号信息，能显著提升游戏交互能力
See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay
- 用视觉帧+符号表示替代纯视觉输入，增强动作决策的准确性
- 真实符号信息使所有模型性能提升，但自动生成符号效果依赖模型能力
- 符号提取可靠性是关键瓶颈，适合研究具身智能与多模态决策的团队
视觉语言模型（VLMs）在描述视觉场景方面表现优异，但在将感知转化为精确、可执行的动作时仍存在困难。本文探究在提供视觉帧的同时加入场景的符号化表示，是否能提升VLM在交互环境中的表现。我们在Atari游戏、VizDoom和AI2-THOR三个平台上评估了三种先进VLMs，对比了仅使用帧、帧+自提取符号、帧+真实符号以及仅符号四种处理流程。结果表明，当符号信息准确时，所有模型性能均有提升；但若由VLM自行提取符号，则性能受模型能力与场景复杂度影响显著。进一步分析发现，VLM从视觉输入中提取符号的能力有限，且符号噪声会严重影响决策与游戏表现。研究揭示：只有在符号提取可靠的前提下，符号化接地才对VLM有益，凸显感知质量是未来基于VLM的智能体发展的核心瓶颈。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines. Our results indicate that all models benefit when the symbolic information is accurate. However, when VLMs extract symbols themselves, performance becomes dependent on model capability and scene complexity. We further investigate how accurately VLMs can extract symbolic information from visual inputs and how noise in these symbols affects decision-making and gameplay performance. Our findings reveal that symbolic grounding is beneficial in VLMs only when symbol extraction is reliable, and highlight perception quality as a central bottleneck for future VLM-based agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。