用文本反转技术让检测模型学新对象,不丢原有能力且只需少量样本。
Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting
- 通过学习新词嵌入扩展模型词汇,仅需三张图即可精准识别新物体。
- 保持原始模型权重冻结,零样本能力与基准性能完全保留。
- 计算量极小,适合快速适配新任务,尤其适合资源有限场景。
大型预训练视觉语言模型(VLM)在多个目标检测基准上达到顶尖水平,具备强大的零样本能力,但针对特定目标仍需微调。尽管初始VLM权重支持出色的少样本迁移学习,但通常会损失原有的自然语言查询和零样本能力。受文本反转(TI)在文生图扩散模型个性化中的成功启发,我们提出一种适用于开放词汇目标检测的类似方法。TI通过学习新词或改进现有词嵌入,实现对新物体或细粒度类别的准确检测,仅需三例样本。所学词嵌入与原始VLM权重完全兼容,模型权重保持冻结,从而保留原模型的基准性能及零样本域迁移能力(如仅用真实照片训练即可检测草图)。存储与梯度计算仅限于词嵌入维度,所需计算量远低于全模型微调。我们在多种定量与定性实验中验证该方法是否在避免遗忘的前提下优于基线方法。
原文摘要 · Abstract (English)
Recent progress in large pre-trained vision language models (VLMs) has reached state-of-the-art performance on several object detection benchmarks and boasts strong zero-shot capabilities, but for optimal performance on specific targets some form of finetuning is still necessary. While the initial VLM weights allow for great few-shot transfer learning, this usually involves the loss of the original natural language querying and zero-shot capabilities. Inspired by the success of Textual Inversion (TI) in personalizing text-to-image diffusion models, we propose a similar formulation for open-vocabulary object detection. TI allows extending the VLM vocabulary by learning new or improving existing tokens to accurately detect novel or fine-grained objects from as little as three examples. The learned tokens are completely compatible with the original VLM weights while keeping them frozen, retaining the original model's benchmark performance, and leveraging its existing capabilities such as zero-shot domain transfer (e.g., detecting a sketch of an object after training only on real photos). The storage and gradient calculations are limited to the token embedding dimension, requiring significantly less compute than full-model fine-tuning. We evaluated whether the method matches or outperforms the baseline methods that suffer from forgetting in a wide variety of quantitative and qualitative experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。