无需标注数据，就能预测视觉语言模型在新任务上的零样本表现。
Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
- 结合文本与生成图像评估模型零样本能力
- 生成图像使预测准确率显著提升
- 帮助非专家用户判断模型是否适用
视觉语言模型（如CLIP）通过对齐图文嵌入空间，使用户仅凭命名类别即可构建视觉分类器。然而，模型在某一领域表现良好未必适用于其他场景，非专家难以判断其适用性。本文基于仅用文本的评估方法，引入与任务相关的合成图像来改进零样本性能预测。实验表明，加入生成图像后，预测准确率显著提升；同时为用户提供评估依据——哪些图像被用于判断模型表现。在标准CLIP基准数据集上的实验验证了该方法在无标签情况下有效预测模型适用性的能力。
原文摘要 · Abstract (English)
Vision-Language Models like CLIP create aligned embedding spaces for text and images, making it possible for anyone to build a visual classifier by simply naming the classes they want to distinguish. However, a model that works well in one domain may fail in another, and non-expert users have no straightforward way to assess whether their chosen VLM will work on their problem. We build on prior work using text-only comparisons to evaluate how well a model works for a given natural language task, and explore approaches that also generate synthetic images relevant to that task to evaluate and refine the prediction of zero-shot accuracy. We show that generated imagery to the baseline text-only scores substantially improves the quality of these predictions. Additionally, it gives a user feedback on the kinds of images that were used to make the assessment. Experiments on standard CLIP benchmark datasets demonstrate that the image-based approach helps users predict, without any labeled examples, whether a VLM will be effective for their application.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。