用9×9网格增强图像空间感知，显著提升多模态模型定位精度。
Grid-augmented vision: A simple yet effective approach for enhanced spatial understanding in multi-modal agents
- 在输入图像上叠加9×9网格，提供显式视觉位置编码。
- 在COCO 2017上，IoU提升107.4%（0.27→0.56），GIoU提升194.4%（0.18→0.53）。
- 方法简单高效，适合机器人操作、自动驾驶等需要精准空间推理的场景。
近年来多模态模型在物体识别和场景理解方面表现出色，但在精确空间定位方面仍存在困难，这制约了其在真实场景中的应用。受人类使用棋盘、地图等网格参考的启发，本文提出一种简单的网格叠加方法，在输入图像上添加9×9黑色网格图案，以提供显式的视觉位置编码，类似于Transformer中的位置编码，但以可视形式呈现。在COCO 2017数据集上的实验表明，该方法显著提升了定位精度：与基线相比，IoU提升107.4%（从0.27增至0.56），GIoU提升194.4%（从0.18增至0.53）。通过注意力可视化分析，验证了该编码有助于模型更准确地捕捉空间关系。该方法结构简单、效果显著，尤其适用于需要高精度空间推理的任务，如机器人抓取、医学影像分析和自动驾驶。
原文摘要 · Abstract (English)
Recent advances in multimodal models have demonstrated impressive capabilities in object recognition and scene understanding. However, these models often struggle with precise spatial localization - a critical capability for real-world applications. Inspired by how humans use grid-based references like chess boards and maps, we propose introducing explicit visual position encoding through a simple grid overlay approach. By adding a 9x9 black grid pattern onto input images, our method provides visual spatial guidance analogous to how positional encoding works in transformers, but in an explicit, visual form. Experiments on the COCO 2017 dataset demonstrate that our grid-based approach achieves significant improvements in localization accuracy, with a 107.4% increase in IoU (from 0.27 to 0.56) and a 194.4% improvement in GIoU (from 0.18 to 0.53) compared to baseline performance. Through attention visualization analysis, we show how this visual position encoding helps models better ground spatial relationships. Our method's simplicity and effectiveness make it particularly valuable for applications requiring accurate spatial reasoning, such as robotic manipulation, medical imaging, and autonomous navigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。