用文本蒸馏让模型更准定位物体,无需框标注。
TeD-Loc: Text Distillation for Weakly Supervised Object Localization
- 通过对比对齐将文本特征迁移到图像块,实现像素级定位
- 在CUB和ILSVRC上定位准确率提升约5%,病理数据集PxAP提升31%
- 比复杂方法更高效,适合追求精度与速度的视觉定位任务
弱监督目标定位(WSOL)模型仅使用图像级类别标签训练,可同时预测物体类别与空间位置,无需边界框标注。传统方法依赖分类目标,常聚焦最具判别性的局部区域,忽略完整空间范围。尽管视觉语言模型如CLIP包含丰富语义先验,但其全局文本与类别令牌嵌入未显式对齐局部图像块嵌入,难以直接用于定位。近期方法如GenPrompt虽解决此问题,但引入条件去噪与复杂提示学习,增加计算负担。本文提出TeD-Loc,通过对比对齐将CLIP文本嵌入知识蒸馏至图像块嵌入,实现块级前景/背景定位。同时引入定位引导的分类模块,利用定位得分聚合前景块嵌入,实现分类与定位联合优化。此外,在蒸馏前采用基于QR的正交化处理类别文本嵌入,增强语义相近类别的区分能力。大量实验表明,TeD-Loc在CUB和ILSVRC上提升约5%的Top-1 Loc,在病理学基准上提升约31%的PxAP,且推理效率优于GenPrompt。
原文摘要 · Abstract (English)
Weakly supervised object localization (WSOL) models are trained using only image-level class labels. They can predict both the object class and spatial regions corresponding to the object, without requiring explicit bounding box annotations. Given their reliance on classification objectives, traditional WSOL methods, like class activation mapping, tend to focus on the most discriminative object regions, often missing the full spatial extent. Although vision-language models such as CLIP encode rich semantic priors, they are not directly suited for WSOL because global text and class-token embeddings are not explicitly aligned with local patch embeddings, making patch-level localization difficult without additional mechanisms. Recent methods such as GenPrompt address this limitation, but at the cost of increased complexity, as they rely on conditional denoising and elaborate prompt-learning strategies. We propose Text Distillation for Localization (TeD-Loc), which transfers knowledge from CLIP text embeddings to patch embeddings through contrastive alignment, thereby enabling patch-level foreground/background localization. A localization-guided classification module is also introduced that uses localization scores to aggregate foreground patch embeddings for joint classification and localization in a single model. In addition, a QR-based orthogonalization of class text embeddings is applied before distillation to improve discrimination for semantically similar classes. Extensive experiments show that TeD-Loc improves Top-1 Loc by ~5% on CUB and ILSVRC, and PxAP by ~31% on histopathology benchmarks, while achieving more efficient inference than GenPrompt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。