arXiv:2603.07911cs.CV2026-03中稿 · CVPR

用贝叶斯框架融合概念引导,提升零样本图像识别准确率

Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition

  • 将类别概念视为潜在变量,构建贝叶斯推理框架
  • 在ImageNet上达84.3%准确率,优于CLIP等主流方法
  • 适合需要高鲁棒性零样本识别的场景

视觉语言模型(如CLIP)虽推动了零样本图像识别发展,但受限于提示工程不佳和对目标类适应性差。现有方法依赖启发式提示设计,缺乏通用性且易受异常提示影响。本文通过引入类别特有概念,将零样本分类重构为概念空间上的边缘化推断,每个概念由先验和图像条件似然加权。为构建高效表达的提议分布,采用大模型驱动的多阶段概念合成流程生成判别性强、可组合的概念,并用行列式点过程保证多样性。针对异常概念干扰,提出无需训练的自适应软剪枝似然,在一次前向传播中削弱其影响。理论分析提供鲁棒性保障与多类超出风险界。大量实验表明,该方法在多个数据集上持续超越现有最优方法。代码已开源。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs), such as CLIP, have significantly advanced zero-shot image recognition. However, their performance remains limited by suboptimal prompt engineering and poor adaptability to target classes. While recent methods attempt to improve prompts through diverse class descriptions, they often rely on heuristic designs, lack versatility, and are vulnerable to outlier prompts. This paper enhances prompt by incorporating class-specific concepts. By treating concepts as latent variables, we rethink zero-shot image classification from a Bayesian perspective, casting prediction as marginalization over the concept space, where each concept is weighted by a prior and a test-image conditioned likelihood. This formulation underscores the importance of both a well-structured concept proposal distribution and the refinement of concept priors. To construct an expressive and efficient proposal distribution, we introduce a multi-stage concept synthesis pipeline driven by LLMs to generate discriminative and compositional concepts, followed by a Determinantal Point Process to enforce diversity. To mitigate the influence of outlier concepts, we propose a training-free, adaptive soft-trim likelihood, which attenuates their impact in a single forward pass. We further provide robustness guarantees and derive multi-class excess risk bounds for our framework. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches, validating its effectiveness in zero-shot image classification. Our code is available at https://github.com/less-and-less-bugs/CGBC.

零样本识别贝叶斯推理提示工程视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。