用视觉大模型提升轻量检测器性能,推理更快更准
RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
- 通过深度语义注入和梯度自适应调制,融合大模型特征
- 在78~273帧/秒下实现57.0的AP最高精度
- 无需增加计算开销,适合移动端实时检测
实时目标检测虽因精心设计的架构和优化策略取得显著进展,但追求高速推理的轻量化网络常导致特征表达能力下降,制约性能提升与设备端部署。本文提出一种低成本、高适应性的知识蒸馏框架,利用快速演进的视觉基础模型(VFMs)增强轻量级检测器。针对VFMs与资源受限检测器间存在显著架构与学习目标差异的问题,我们引入深度语义注入(DSI)模块,将大模型高层语义信息融入检测器深层;同时设计基于梯度范数比的自适应调制(GAM)策略,动态调节语义传递强度。该方法无额外部署与推理开销,在多种DETR基线模型上实现稳定且显著的性能提升。新提出的RT-DETRv4系列在COCO数据集上达到49.7/53.5/55.4/57.0的AP值,对应推理速度分别为273/169/124/78 FPS。
原文摘要 · Abstract (English)
Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a Deep Semantic Injector (DSI) module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a Gradient-guided Adaptive Modulation (GAM) strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of 49.7/53.5/55.4/57.0 at corresponding speeds of 273/169/124/78 FPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。