arXiv:2503.07465cs.CV2025-03ICCV被引 69

YOLOE让模型实时识别任意物体,支持文本、视觉和无提示三种方式。

YOLOE: Real-Time Seeing Anything

  • 统一框架融合三类开放提示,实现高效多模态感知。
  • 在LVIS上训练成本降3倍,推理速度提升1.4倍,性能领先3.5点AP。
  • 适合需要快速部署、低资源开销的开放世界检测场景。

目标检测与分割广泛应用于计算机视觉,但传统YOLO系列模型受限于预定义类别,在开放场景下适应性差。现有开放集方法虽利用文本提示、视觉线索或免提示范式突破限制,却常因计算开销大或部署复杂而牺牲性能与效率。本文提出YOLOE,将检测与分割统一于单一高效模型中,实现真正的实时‘看见任何东西’。针对文本提示,提出可重参数化区域-文本对齐(RepRTA)策略,通过轻量辅助网络优化预训练文本嵌入,实现零推理开销的视觉-文本对齐。针对视觉提示,设计解耦语义与激活分支的语义激活视觉提示编码器(SAVPE),在极低复杂度下提升视觉嵌入质量与精度。针对免提示场景,引入懒惰区域提示对比(LRPC)策略,内置大词表与专用嵌入,无需依赖语言模型即可识别所有物体。大量实验表明,YOLOE在零样本性能与迁移能力上表现优异,兼具高推理效率与低训练成本。尤其在LVIS数据集上,YOLOE-v8-S以3倍更低训练成本、1.4倍更快推理速度,超越YOLO-Worldv2-S达3.5 AP;迁移至COCO时,YOLOE-v8-L相较闭集YOLOv8-L分别取得0.6 AP^b与0.4 AP^m提升,训练时间仅需其约四分之一。代码与模型已开源:https://github.com/THU-MIG/yoloe。

原文摘要 · Abstract (English)

Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE's exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with 3$\times$ less training cost and 1.4$\times$ inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 AP$^b$ and 0.4 AP$^m$ gains over closed-set YOLOv8-L with nearly 4$\times$ less training time. Code and models are available at https://github.com/THU-MIG/yoloe.

目标检测开放世界实时识别多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。