让语言模型更准定位图像物体，通过可学习的网格与偏移令牌。
Grounding Everything in Tokens for Multimodal Large Language Models
- 引入可学习的网格与偏移令牌，构建2D空间定位机制。
- 在多个指代任务中超越现有方法，提升定位精度。
- 无需改动自回归结构，适合主流多模态大模型部署。
多模态大语言模型（MLLMs）在视觉理解与推理方面取得显著进展，但其采用的自回归Transformer架构需对输入图像进行分词处理，限制了其在二维图像空间中精确定位物体的能力。如何改进序列语言令牌以更好实现物体在2D空间中的定位成为关键问题。为此，我们提出一种名为GETok的空间表示方法，将可学习的专用令牌融入MLLMs。GETok首先使用网格令牌将图像平面划分为结构化空间锚点，再利用偏移令牌实现定位预测的精确且迭代式优化。通过将空间关系直接嵌入令牌，GETok在不改变自回归架构的前提下，显著提升了模型在原生2D空间中的推理能力。大量实验表明，GETok在监督微调与强化学习设置下，于多个指代任务中均优于当前最优方法。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have made significant advancements in vision understanding and reasoning. However, the autoregressive Transformer architecture used by MLLMs requries tokenization on input images, which limits their ability to accurately ground objects within the 2D image space. This raises an important question: how can sequential language tokens be improved to better ground objects in 2D spatial space for MLLMs? To address this, we present a spatial representation method for grounding objects, namely GETok, that integrates a specialized vocabulary of learnable tokens into MLLMs. GETok first uses grid tokens to partition the image plane into structured spatial anchors, and then exploits offset tokens to enable precise and iterative refinement of localization predictions. By embedding spatial relationships directly into tokens, GETok significantly advances MLLMs in native 2D space reasoning without modifying the autoregressive architecture. Extensive experiments demonstrate that GETok achieves superior performance over the state-of-the-art methods across various referring tasks in both supervised fine-tuning and reinforcement learning settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。