让AI像人一样反复看图推理，提升多模态思考准确性
v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning
- 通过选点复制机制，动态召回图像关键区域嵌入
- 在数学推理任务上，准确率比基线最高提升12.3%
- 适合需要精细视觉推理的复杂任务场景
人类思考图像时不会只看一眼，而是会反复回看视觉信息。相比之下，大多数多模态语言模型仅将图像编码一次存入键值缓存，后续推理完全依赖文本，难以在中间步骤重新定位视觉证据。我们实证发现：随着推理链变长，模型对关键区域的关注度逐渐下降。为此提出v1，一种轻量级扩展，通过‘选点复制’实现主动视觉引用：模型选择相关图像块，并将其嵌入向量复制回推理流。关键在于，该机制使用语义表示作为键来检索图像块，确保感知证据与推理空间保持对齐。为训练此行为，我们构建了v1g数据集，包含30万条带有交错标注的多模态推理轨迹。在多个多模态数学推理基准上，v1持续优于可比基线。代码、模型与数据均已开源。
原文摘要 · Abstract (English)
When thinking with images, humans rarely rely on a single glance: they revisit visual evidence while reasoning. In contrast, most Multimodal Language Models encode an image once to key-value cache and then reason purely in text, making it hard to re-ground intermediate steps. We empirically confirm this: as reasoning chains lengthen, models progressively lose focus on relevant regions. We introduce v1, a lightweight extension for active visual referencing via point-and-copy: the model selects relevant image patches and copies their embeddings back into the reasoning stream. Crucially, our point-and-copy mechanism retrieves patches using their semantic representations as keys, ensuring perceptual evidence remains aligned with the reasoning space. To train this behavior, we build v1g, a dataset of 300K multimodal reasoning traces with interleaved grounding annotations. Across multimodal mathematical reasoning benchmarks, v1 consistently outperforms comparable baselines. We release our code, model, and data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。