让视觉语言模型实时检测多类别目标，速度提升25倍
Detect Anything in Real Time: From Single-Prompt Segmentation to Multi-Class Detection
- 共享图像主干特征，批量处理多类查询
- 80类检测达15.8帧/秒，精度超专用模型
- 无需训练，适配高实时性场景的部署
近期视觉-语言建模进展催生了可接受任意自然语言查询的可提示检测与分割系统。其中，SAM3通过使用ViT-H/14主干网络、跨模态Transformer解码器和学习过的对象查询，实现了最先进的准确率。然而，SAM3每次前向传播仅处理一个文本提示，检测N个类别需执行N次独立推理，且每次均受439M参数主干网络主导。本文提出Detect Anything in Real Time（DART），一种无需训练的框架，通过利用结构不变性——视觉主干网络对类别无关，生成不依赖文本提示的图像特征——实现主干计算在所有类别间共享，使主干开销从O(N)降至O(1)。结合批处理多类解码、仅检测推理和TensorRT FP16部署，优化后在3类任务上获得5.6倍累计加速，80类时达25倍，且未修改任何模型权重。在COCO val2017数据集（5,000张图像，80类）上，DART以15.8 FPS（4类，1008×1008）达到55.8 AP，超越基于数百万边界框标注训练的专用开放词汇检测器。对于极低延迟需求，采用冻结编码器-解码器的适配器蒸馏方案，在13.9毫秒内完成主干计算，精度为38.7 AP。代码与模型已公开于https://github.com/mkturkcan/DART。
原文摘要 · Abstract (English)
Recent advances in vision-language modeling have produced promptable detection and segmentation systems that accept arbitrary natural language queries at inference time. Among these, SAM3 achieves state-of-the-art accuracy by combining a ViT-H/14 backbone with cross-modal transformer decoding and learned object queries. However, SAM3 processes a single text prompt per forward pass. Detecting N categories requires N independent executions, each dominated by the 439M-parameter backbone. We present Detect Anything in Real Time (DART), a training-free framework that converts SAM3 into a real-time multi-class detector by exploiting a structural invariant: the visual backbone is class-agnostic, producing image features independent of the text prompt. This allows the backbone computation to be shared between all classes, reducing its cost from O(N) to O(1). Combined with batched multi-class decoding, detection-only inference, and TensorRT FP16 deployment, these optimizations yield 5.6x cumulative speedup at 3 classes, scaling to 25x at 80 classes, without modifying any model weight. On COCO val2017 (5,000 images, 80 classes), DART achieves 55.8 AP at 15.8 FPS (4 classes, 1008x1008) on a single RTX 4080, surpassing purpose-built open-vocabulary detectors trained on millions of box annotations. For extreme latency targets, adapter distillation with a frozen encoder-decoder achieves 38.7 AP with a 13.9 ms backbone. Code and models are available at https://github.com/mkturkcan/DART.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。