用语言模型迭代优化视觉分类，让零样本识别更准。
Generate, Transduct, Adapt: Iterative Transduction with VLMs
- 三步迭代：查语言模型扩属性、加属性推理、用推断标签微调
- 零样本下比CLIP平均高8.6%，比传统迁移方法高3.7%
- 适合做零样本/少样本视觉分类的研究者和应用者
基于视觉-语言模型的归纳式零样本学习利用数据集内图像间的相似性，可获得比归纳设置更高的分类准确率。然而，现有研究很少关注语言空间在此情境下的结构。本文提出GTA-CLIP，一种结合语言模型监督的联合语言与视觉空间的迭代式迁移方法。该方法包含三个步骤：(i) 通过查询语言模型逐步探索属性空间；(ii) 基于属性增强的迁移推理；(iii) 利用数据集内推断标签对语言和视觉编码器进行微调。在使用CLIP编码器的实验中，GTA-CLIP在12个数据集上平均比CLIP提升8.6%，比传统迁移方法提升3.7%。在少样本设置下也观察到类似改进。消融实验证明各步骤价值，并可视化了视觉与语言空间随迭代演化的动态过程。代码已开源。
原文摘要 · Abstract (English)
Transductive zero-shot learning with vision-language models leverages image-image similarities within the dataset to achieve better classification accuracy compared to the inductive setting. However, there is little work that explores the structure of the language space in this context. We propose GTA-CLIP, a novel technique that incorporates supervision from language models for joint transduction in language and vision spaces. Our approach is iterative and consists of three steps: (i) incrementally exploring the attribute space by querying language models, (ii) an attribute-augmented transductive inference procedure, and (iii) fine-tuning the language and vision encoders based on inferred labels within the dataset. Through experiments with CLIP encoders, we demonstrate that GTA-CLIP, yields an average performance improvement of 8.6% and 3.7% across 12 datasets and 3 encoders, over CLIP and transductive CLIP respectively in the zero-shot setting. We also observe similar improvements in a few-shot setting. We present ablation studies that demonstrate the value of each step and visualize how the vision and language spaces evolve over iterations driven by the transductive learning. Code is released at https://github.com/cvl-umass/GTA-CLIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。