用大模型生成精准图文提示,提升少样本学习效果
VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning
- 用大模型结合图像生成精确类别描述,实现跨模态推理
- 在10个不同数据集上达到新最优,跨域与细粒度任务均表现优异
- 适合研究少样本学习、多模态融合与大模型应用的学者
少样本学习(FSL)旨在仅凭少量标注样本识别新概念。现有方法通过引入语义信息或设计复杂融合模块来增强支持特征,但因缺乏真实实例的语义锚定,常产生与视觉证据矛盾的幻觉,导致噪声引导和高成本修正。为此,本文提出一种新框架VT-FSL,通过大语言模型(LLM)与支持图像构建精确的跨模态提示,并经几何感知对齐实现无缝整合。核心包含交叉模态迭代提示(CIP)与交叉模态几何对齐(CGA)。CIP在单次结构化推理中,基于类别名和支持图像条件化LLM,迭代生成精准类别描述;这些描述既丰富了新类别的语义理解,又可零样本合成语义一致的图像。描述与合成图像分别作为互补的文本与视觉提示,提供高层语义与低层类内多样性,弥补支持数据不足。CGA通过最小化三元组平行体的核体积,联合对齐融合后的文本、支持图像与合成视觉表示,捕捉全局非线性关系,实现结构化一致的多模态融合。VT-FSL在十个多样化基准上均取得新最佳性能,涵盖标准、跨域与细粒度少样本学习场景。代码已开源。
原文摘要 · Abstract (English)
Few-shot learning (FSL) aims to recognize novel concepts from only a few labeled support samples. Recent studies enhance support features by incorporating additional semantic information or designing complex semantic fusion modules. However, they still suffer from hallucinating semantics that contradict the visual evidence due to the lack of grounding in actual instances, resulting in noisy guidance and costly corrections. To address these issues, we propose a novel framework, bridging Vision and Text with LLMs for Few-Shot Learning (VT-FSL), which constructs precise cross-modal prompts conditioned on Large Language Models (LLMs) and support images, seamlessly integrating them through a geometry-aware alignment. It mainly consists of Cross-modal Iterative Prompting (CIP) and Cross-modal Geometric Alignment (CGA). Specifically, the CIP conditions an LLM on both class names and support images to generate precise class descriptions iteratively in a single structured reasoning pass. These descriptions not only enrich the semantic understanding of novel classes but also enable the zero-shot synthesis of semantically consistent images. The descriptions and synthetic images act respectively as complementary textual and visual prompts, providing high-level class semantics and low-level intra-class diversity to compensate for limited support data. Furthermore, the CGA jointly aligns the fused textual, support, and synthetic visual representations by minimizing the kernelized volume of the 3-dimensional parallelotope they span. It captures global and nonlinear relationships among all representations, enabling structured and consistent multimodal integration. The proposed VT-FSL method establishes new state-of-the-art performance across ten diverse benchmarks, including standard, cross-domain, and fine-grained few-shot learning scenarios. Code is available at https://github.com/peacelwh/VT-FSL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。