arXiv:2512.17160cs.CV2025-12中稿 · IEEE ICASSP2026

用大模型生成图像原型,零样本分类无需人工标注图文对。

Can Synthetic Images Serve as Effective and Efficient Class Prototypes?

  • 用大语言模型生成类别提示词,驱动扩散模型合成视觉原型图。
  • 仅需类别标签即可完成训练与推理,零样本分类准确率达78.2%。
  • 只需视觉编码器,模型轻量高效,适合资源受限场景使用。

视觉语言模型在零样本图像分类任务中表现优异,但现有方法如对比语言-图像预训练(CLIP)均依赖标注的图文配对来对齐模态,导致数据准备成本高且质量要求严苛。同时,双塔编码器结构也增加了计算负担。为此,本文提出基于大语言模型生成的对比语言-图像预训练框架(LGCLIP)。该框架利用大语言模型生成特定类别的提示词,指导扩散模型合成参考图像作为视觉原型。随后,真实图像的视觉特征与这些生成原型的特征进行比对以实现分类预测。通过优化提示生成过程并仅使用视觉编码器,LGCLIP保持轻量化与高效性。关键在于,整个实验流程仅需类别标签输入,无需人工标注图文对或额外预处理。实验验证了该方法的可行性和效率,在多个零样本分类任务中表现优异,建立了新的分类范式。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have shown strong performance in zero-shot image classification tasks. However, existing methods, including Contrastive Language-Image Pre-training (CLIP), all rely on annotated text-to-image pairs for aligning visual and textual modalities. This dependency introduces substantial cost and accuracy requirement in preparing high-quality datasets. At the same time, processing data from two modes also requires dual-tower encoders for most models, which also hinders their lightweight. To address these limitations, we introduce a ``Contrastive Language-Image Pre-training via Large-Language-Model-based Generation (LGCLIP)" framework. LGCLIP leverages a Large Language Model (LLM) to generate class-specific prompts that guide a diffusion model in synthesizing reference images. Afterwards these generated images serve as visual prototypes, and the visual features of real images are extracted and compared with the visual features of these prototypes to achieve comparative prediction. By optimizing prompt generation through the LLM and employing only a visual encoder, LGCLIP remains lightweight and efficient. Crucially, our framework requires only class labels as input during whole experimental procedure, eliminating the need for manually annotated image-text pairs and extra pre-processing. Experimental results validate the feasibility and efficiency of LGCLIP, demonstrating great performance in zero-shot classification tasks and establishing a novel paradigm for classification.

零样本学习图像生成大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。