用新点预测方法实现零样本通用目标检测,性能媲美传统模型。
Detect Anything via Next Point Prediction
- 将坐标预测转为离散点选择,降低学习难度并提升效率。
- 在COCO和LVIS上零样本性能超越或媲美YOLO、Grounding DINO等模型。
- 支持指物、视觉提示、界面定位等多任务,适合语言感知系统研究者。
目标检测长期依赖基于坐标回归的模型(如YOLO、DETR、Grounding DINO)。尽管近期尝试利用多模态大模型(MLLM)解决此问题,但仍面临召回率低、重复预测、坐标错位等挑战。本文提出Rex-Omni,一个30亿参数规模的MLLM,在零样本设置下于COCO和LVIS等基准上达到或超过传统回归模型的性能。其关键设计包括:1)任务建模:使用0-999的量化坐标特殊标记,降低学习难度并提升令牌效率;2)数据引擎:构建多个数据引擎生成高质量指物、参照与指向数据,提供语义丰富的监督信号;3)训练流程:采用两阶段训练,先在2200万条数据上进行监督微调,再通过基于GRPO的强化学习后训练,利用几何感知奖励弥合离散到连续坐标的鸿沟,提升框精度并抑制重复预测等教师引导带来的不良行为。除常规检测外,Rex-Omni还具备指物、视觉提示、GUI接地、空间参照、OCR与关键点定位等能力,并在专用基准上系统评估。我们相信Rex-Omni为更通用、语言感知的视觉感知系统开辟了新路径。
原文摘要 · Abstract (English)
Object detection has long been dominated by traditional coordinate regression-based models, such as YOLO, DETR, and Grounding DINO. Although recent efforts have attempted to leverage MLLMs to tackle this task, they face challenges like low recall rate, duplicate predictions, coordinate misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a 3B-scale MLLM that achieves state-of-the-art object perception performance. On benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot setting. This is enabled by three key designs: 1) Task Formulation: we use special tokens to represent quantized coordinates from 0 to 999, reducing the model's learning difficulty and improving token efficiency for coordinate prediction; 2) Data Engines: we construct multiple data engines to generate high-quality grounding, referring, and pointing data, providing semantically rich supervision for training; \3) Training Pipelines: we employ a two-stage training process, combining supervised fine-tuning on 22 million data with GRPO-based reinforcement post-training. This RL post-training leverages geometry-aware rewards to effectively bridge the discrete-to-continuous coordinate prediction gap, improve box accuracy, and mitigate undesirable behaviors like duplicate predictions that stem from the teacher-guided nature of the initial SFT stage. Beyond conventional detection, Rex-Omni's inherent language understanding enables versatile capabilities such as object referring, pointing, visual prompting, GUI grounding, spatial referring, OCR and key-pointing, all systematically evaluated on dedicated benchmarks. We believe that Rex-Omni paves the way for more versatile and language-aware visual perception systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。