arXiv:2505.18039cs.CV2025-05CVPR被引 2

将CLIP模型压缩到车载边缘设备,实现实时图像标注。

Clip4Retrofit: Enabling Real-Time Image Labeling on Edge Devices via Cross-Architecture CLIP Distillation

  • 用轻量级模型蒸馏CLIP,保留跨模态对齐能力。
  • 在计算资源受限的Retrofit相机上实现实时标注。
  • 适合自动驾驶和老旧系统智能化改造场景。

像CLIP(对比语言-图像预训练)这样的基础模型通过跨模态对齐实现了零样本和少样本学习,但其高计算复杂度和大内存占用使其难以部署在资源受限的边缘设备上,如用于图像采集与实时处理的车载摄像头。为此,我们提出Clip4Retrofit,一种高效的模型蒸馏框架,可在边缘设备上实现实时图像标注。该框架部署于Retrofit相机——一种成本低廉且已改装至数千辆汽车的边缘设备,面临严格的算力与内存限制。我们的方法将CLIP知识蒸馏到一个轻量学生模型中,采用EfficientNet-B3结合多层感知机(MLP)投影头,在显著降低计算需求的同时保持跨模态对齐。实验表明,所提出的模型在效率与性能间取得平衡,可在资源受限的边缘设备上完成实时图像标注与物体识别,为自动驾驶及现有系统智能化改造提供可行方案。本工作弥合了前沿视觉-语言模型与资源受限环境部署之间的鸿沟,推动基础模型在边缘计算中的广泛应用。

原文摘要 · Abstract (English)

Foundation models like CLIP (Contrastive Language-Image Pretraining) have revolutionized vision-language tasks by enabling zero-shot and few-shot learning through cross-modal alignment. However, their computational complexity and large memory footprint make them unsuitable for deployment on resource-constrained edge devices, such as in-car cameras used for image collection and real-time processing. To address this challenge, we propose Clip4Retrofit, an efficient model distillation framework that enables real-time image labeling on edge devices. The framework is deployed on the Retrofit camera, a cost-effective edge device retrofitted into thousands of vehicles, despite strict limitations on compute performance and memory. Our approach distills the knowledge of the CLIP model into a lightweight student model, combining EfficientNet-B3 with multi-layer perceptron (MLP) projection heads to preserve cross-modal alignment while significantly reducing computational requirements. We demonstrate that our distilled model achieves a balance between efficiency and performance, making it ideal for deployment in real-world scenarios. Experimental results show that Clip4Retrofit can perform real-time image labeling and object identification on edge devices with limited resources, offering a practical solution for applications such as autonomous driving and retrofitting existing systems. This work bridges the gap between state-of-the-art vision-language models and their deployment in resource-constrained environments, paving the way for broader adoption of foundation models in edge computing.

边缘计算模型蒸馏CLIP实时标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。