让模型能准确识别任意描述中的人，提升真实场景下的实用能力。
Referring to Any Person
- 结合大语言模型与目标检测，构建可理解复杂描述的多主体识别模型
- 新数据集HumanRef涵盖多人共现场景，测试效果显著优于现有模型
- 不仅擅长人物定位，还能泛化到普通物体识别，适用范围广
人类是计算机视觉中最重要的参与者。根据自然语言描述精准定位任意个体的能力——我们称之为「任意人物指代」——具有重要实际价值。然而，现有模型普遍难以满足真实应用需求，且当前基准多聚焦于一对一指代，限制了该方向发展。本文从任务定义、数据集设计和模型架构三方面重新审视此任务：首先提出五类可指代实体与三项任务特征；其次构建全新数据集HumanRef，更贴近真实场景；最后设计融合多模态大语言模型与目标检测框架的RexSeek模型。实验表明，主流模型在常用基准（如RefCOCO/+/g）上表现良好，但在HumanRef上因无法处理多人而失效；而RexSeek不仅能高效完成人物指代，还具备良好通用性，适用于常见物体指代任务。代码已开源。
原文摘要 · Abstract (English)
Humans are undoubtedly the most important participants in computer vision, and the ability to detect any individual given a natural language description, a task we define as referring to any person, holds substantial practical value. However, we find that existing models generally fail to achieve real-world usability, and current benchmarks are limited by their focus on one-to-one referring, that hinder progress in this area. In this work, we revisit this task from three critical perspectives: task definition, dataset design, and model architecture. We first identify five aspects of referable entities and three distinctive characteristics of this task. Next, we introduce HumanRef, a novel dataset designed to tackle these challenges and better reflect real-world applications. From a model design perspective, we integrate a multimodal large language model with an object detection framework, constructing a robust referring model named RexSeek. Experimental results reveal that state-of-the-art models, which perform well on commonly used benchmarks like RefCOCO/+/g, struggle with HumanRef due to their inability to detect multiple individuals. In contrast, RexSeek not only excels in human referring but also generalizes effectively to common object referring, making it broadly applicable across various perception tasks. Code is available at https://github.com/IDEA-Research/RexSeek
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。