arXiv:2510.14583cs.CVcs.CL2025-10

让模型用自然语言精准定位图像像素点,突破传统对象级理解局限。

Talking Points: Describing and Localizing Pixels

  • 提出双向框架:先生成上下文描述,再回归精确坐标。
  • 在2万+三元组数据集上实现优于基线的像素级定位准确率。
  • 适合需要精细定位的视觉理解任务,如医学影像分析。

视觉-语言模型在跨模态理解上取得显著进展,但仍局限于对象级或区域级定位,缺乏通过自然语言进行像素级关键点理解的能力。本文提出一种全新的像素级定位框架,包含两个互补组件:点描述器(Point Descriptor)生成丰富、上下文相关的关键点描述,点定位器(Point Localizer)则从这些描述中回归出精确像素坐标。不同于依赖模板提示或关键点名称的先前方法,本方法生成自由形式、由粗到精的描述,将关键点置于其视觉上下文中。由于缺乏可用训练数据,我们构建了LlamaPointInPart数据集,包含20,000多个由多模型合成的图像-关键点-描述三元组,涵盖从场景级上下文到关键点周围视觉特征的多尺度信息。为实现跨类别泛化,我们在AP-10K上使用GRPO优化点描述器,以冻结的点定位器作为奖励模型,生成能最大化定位准确性的描述。为评估结果,我们建立新评估协议:不比较文本描述与真实描述,而是用定位器判断预测点与真实点的距离。实验表明,该方法在LlamaPointInPart上表现优于基线模型。框架的双向特性为未来关键点引导图像理解与语言引导精准定位应用提供了可能。代码与数据集已开源。

原文摘要 · Abstract (English)

Vision-language models have achieved remarkable success in cross-modal understanding. Yet, these models remain limited to object-level or region-level grounding, lacking the capability for pixel-precise keypoint comprehension through natural language. We introduce a novel framework for pixel level grounding. The framework consists of two complementary components: a Point Descriptor that generates rich, contextual descriptions of individual keypoints, and a Point Localizer that regresses precise pixel coordinates from these descriptions. Unlike prior work that relies on templated prompts or keypoint names, our approach produces free-form, coarse-to-fine descriptions that situate keypoints within their visual context. Since there is no available dataset to train such a system, we introduce LlamaPointInPart, a carefully curated dataset of 20K+ image-keypoint-description triplets synthesized from multiple vision-language models, capturing multi-scale information from scene-level context to visual features around the keypoint. For cross-category generalization, we optimize the Point Descriptor on AP-10K via GRPO, using the frozen Point Localizer as a reward model to produce descriptions that maximize localization accuracy. To evaluate our results we establish a new evaluation protocol. Instead of comparing the text description produced by our method to the ground truth, we use the localizer to determine how close is the predicted point generated to the ground truth point. Experiments demonstrate superior performance compared to baseline models on LlamaPointInPart.The bidirectional nature of our framework should enable future applications in both keypoint-guided image understanding and language-guided precise localization. Our code and dataset are publicly available at https://github.com/matanr/Talking_Points.

像素定位视觉语言关键点描述

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。