用生成模型精调CLIP的候选标签，提升零样本图像分类准确率。
G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification
- 先用CLIP筛选候选标签，再用生成模型逐个验证
- 平均准确率68.85%，比CLIP高9.5个百分点
- 无需训练，适配多种视觉语言模型
零样本分类需要高效标签检索与细粒度视觉推理，但判别式与生成式视觉语言模型存在互补性缺陷。当CLIP的top-1预测错误时，正确标签通常仍存在于其top-K候选列表中，因此关键挑战在于消歧而非召回。独立生成模型则受限于庞大的标签空间和无约束输出。为此，我们提出G2D——一种无需训练的框架，利用生成式VLM对CLIP检索出的候选标签进行图像引导的验证。候选名称与CLIP概率构成结构化先验，用于区分视觉相似类别。通过固定置信度路由、自适应熵候选数调整及字典树约束解码，聚焦不确定样本并确保每张输入仅输出一个有效结果。在八个基准上，G2D平均准确率达68.85%，优于CLIP的59.35%和独立生成模型的63.11%。七种生成器配置下，候选集验证使平均准确率提升1.08至27.42个百分点。G2D还可迁移至DCLIP、WaffleCLIP和CuPL，为判别式提案与生成式视觉推理提供实用接口。
原文摘要 · Abstract (English)
Zero-shot classification needs efficient label retrieval and fine-grained visual reasoning, yet discriminative and generative vision-language models fail in complementary ways.When CLIP's top-1 prediction is wrong, the correct label often remains in its top-$K$ shortlist, making disambiguation rather than recall the key challenge.Standalone generative models, however, are hindered by large label spaces and unconstrained outputs.This complementarity motivates separating broad candidate retrieval from fine-grained, image-grounded verification.We propose G2D, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.Candidate names and CLIP probabilities provide a structured prior for resolving visually similar classes.Fixed confidence routing, entropy-adaptive candidate sizing, and trie-constrained decoding focus generative reasoning on uncertain samples and ensure one valid output for each input at test time.Across eight benchmarks, G2D achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.Across seven generator configurations, candidate-set verification improves average accuracy by 1.08--27.42 percentage points.G2D also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning. Code: https://github.com/Harzva/G2D
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。