arXiv:2509.23746cs.CVcs.AI2025-09被引 3

让AI通过自我修正精准定位图像目标,性能超越谷歌和开源模型。

Poivre: Self-Refining Visual Pointing with Reinforcement Learning

  • 用强化学习驱动AI分步修正定位点,模拟人类反复调整的视觉指认过程。
  • 在Point-Bench上达到新纪录,比Gemini-2.5-Pro和Molmo-72B高出3%以上。
  • 适合研究视觉语言模型、多轮交互式定位与强化学习应用的开发者和研究者。

视觉指认旨在通过预测图像中目标的位置坐标来实现定位,是视觉语言模型(VLMs)的重要任务。尽管应用广泛,现有基准测试显示当前VLMs在该任务上的表现仍远低于人类水平。主要瓶颈在于大多数VLMs需在单步内完成指认,如同要求人类不看手指就准确指向物体。为此,我们提出一种简单而高效的方法:点、可视化、再修正(Poivre)。该方法允许模型先标记初步位置,随后根据需要迭代优化坐标。受自然语言推理模型进展启发,我们采用强化学习(RL)来激励这一自我修正能力。针对RL训练,设计了一种简洁且有效的过程奖励机制,具备良好实证效果与理论合理性。训练得到的Poivre-7B模型在Point-Bench上达到新最优,显著优于包括Gemini-2.5-Pro在内的专有模型及Molmo-72B等开源大模型,性能提升超3%。为支持后续研究,我们公开了训练与推理代码、数据集及Poivre-7B模型权重。

原文摘要 · Abstract (English)

Visual pointing, which aims to localize a target by predicting its coordinates on an image, has emerged as an important problem in the realm of vision-language models (VLMs). Despite its broad applicability, recent benchmarks show that current VLMs still fall far behind human performance on this task. A key limitation is that VLMs are typically required to complete the pointing task in a single step, akin to asking humans to point at an object without seeing their own fingers. To address this issue, we propose a simple yet effective self-refining procedure: Point, Visualize, then Refine (Poivre). This procedure enables a VLM to first mark its estimated point, then iteratively refine the coordinates if necessary. Inspired by advances of reasoning models in the natural language domain, we employ reinforcement learning (RL) to incentivize this self-refining ability. For the RL training, we design a neat process reward that is not only empirically effective but also grounded in appealing properties. Our trained model, Poivre-7B, sets a new state of the art on Point-Bench, outperforming both proprietary models such as Gemini-2.5-Pro and large open-source models such as Molmo-72B by over 3%. To support future research, we release our training and inference code, dataset, and the Poivre-7B checkpoint.

视觉定位强化学习自修正视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。