用视觉语言对比学习提升目标检测准确率
CLIP-Joint-Detect: End-to-End Joint Training of Object Detectors with Contrastive Vision-Language Supervision
- 通过对比学习将图像区域与可学习文本嵌入对齐
- 在Pascal VOC和MS COCO上均实现显著性能提升
- 兼容主流检测框架且保持实时推理速度
传统目标检测器依赖交叉熵分类，易受类别不平衡和标签噪声影响。我们提出CLIP-Joint-Detect，一种简单、检测器无关的端到端联合训练框架，通过对比视觉语言监督集成CLIP式对比学习。一个轻量级并行头将区域或网格特征投影到CLIP嵌入空间，并通过InfoNCE对比损失和辅助交叉熵项与可学习的类别特定文本嵌入对齐，同时优化所有标准检测损失。该方法可无缝应用于两阶段和单阶段架构。我们在Pascal VOC 2007+2012上使用Faster R-CNN，在大规模MS COCO 2017基准上使用现代YOLO检测器（YOLOv11）进行验证，实现了持续且显著的性能提升，同时保持实时推理速度。大量实验与消融分析表明，与可学习文本嵌入联合优化显著提升了多种架构和数据集上的闭集检测性能。
原文摘要 · Abstract (English)
Conventional object detectors rely on cross-entropy classification, which can be vulnerable to class imbalance and label noise. We propose CLIP-Joint-Detect, a simple and detector-agnostic framework that integrates CLIP-style contrastive vision-language supervision through end-to-end joint training. A lightweight parallel head projects region or grid features into the CLIP embedding space and aligns them with learnable class-specific text embeddings via InfoNCE contrastive loss and an auxiliary cross-entropy term, while all standard detection losses are optimized simultaneously. The approach applies seamlessly to both two-stage and one-stage architectures. We validate it on Pascal VOC 2007+2012 using Faster R-CNN and on the large-scale MS COCO 2017 benchmark using modern YOLO detectors (YOLOv11), achieving consistent and substantial improvements while preserving real-time inference speed. Extensive experiments and ablations demonstrate that joint optimization with learnable text embeddings markedly enhances closed-set detection performance across diverse architectures and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。