用区域嵌入增强视觉语言模型,提升跨域图像分类准确率。
LARE: Latent Augmentation using Regional Embedding with Vision-Language Model
- 将图像嵌入为统一空间中的一个区域,而非单点,实现更丰富的表征。
- 在三个基准上超越现有微调方法,尤其在未见领域表现更优。
- 适用于小样本、数据不平衡等挑战场景,适合实际部署的鲁棒性需求。
近年来,视觉语言模型(VLM)在处理图像与文本数据方面取得显著进展,广泛应用于“图像对话”、“指令式图像识别”和“视觉问答”等下游任务。以对比语言-图像预训练(CLIP)为代表的VLM具备高性能图像分类能力,并被拓展为利用语言信息进行领域自适应的方法,以扩展至未见领域。然而,由于这些VLM将图像嵌入为统一嵌入空间中的单一向量,分类精度仍有提升空间。为此,本文提出潜在区域增强方法(Latent Augmentation using Regional Embedding, LARE),将图像嵌入为该空间中的一个区域。通过从该潜在区域内采样增强图像嵌入,LARE可实现对多种未见领域的数据增强。利用这些增强嵌入微调VLM,LARE在域内与域外均实现了稳健的图像分类性能。实验表明,LARE在三个基准测试中优于以往微调模型,在未见领域、小样本和数据不平衡等多种条件下均展现出更强的鲁棒性与泛化能力。
原文摘要 · Abstract (English)
In recent years, considerable research has been conducted on vision-language models that handle both image and text data; these models are being applied to diverse downstream tasks, such as "image-related chat," "image recognition by instruction," and "answering visual questions." Vision-language models (VLMs), such as Contrastive Language-Image Pre-training (CLIP), are also high-performance image classifiers that are being developed into domain adaptation methods that can utilize language information to extend into unseen domains. However, because these VLMs embed images as a single point in a unified embedding space, there is room for improvement in the classification accuracy. Therefore, in this study, we proposed the Latent Augmentation using Regional Embedding (LARE), which embeds the image as a region in the unified embedding space learned by the VLM. By sampling the augmented image embeddings from within this latent region, LARE enables data augmentation to various unseen domains, not just to specific unseen domains. LARE achieves robust image classification for domains in and out using augmented image embeddings to fine-tune VLMs. We demonstrate that LARE outperforms previous fine-tuning models in terms of image classification accuracy on three benchmarks. We also demonstrate that LARE is a more robust and general model that is valid under multiple conditions, such as unseen domains, small amounts of data, and imbalanced data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。