无需训练即可高效适配视觉语言模型，利用动态图传播标签。
Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model
- 构建文本、样本与测试数据的动态图，通过图传播推断标签。
- 在细粒度分类和分布外泛化任务上表现优于现有方法。
- 适合资源受限场景下的零/少样本快速部署。
视觉语言模型（VLMs）通过大规模预训练显著提升了机器学习性能。尽管在标签、训练和数据效率方面已有进步，当前多数先进VLM仍需针对任务调参，且未能充分利用测试样本。为此，我们提出一种基于图的标签高效适配与推理方法。该方法在文本提示、少样本示例和测试样本间动态构建图结构，通过标签传播实现无需任务调参的推理。相比现有零样本标签传播技术，本方法无需额外无标签支持集，并通过动态图扩展有效利用测试样本流形。进一步引入上下文感知特征重加权机制，提升任务适应精度。同时支持高效图扩展，实现实时归纳推理。在细粒度分类与分布外泛化等下游任务上的大量实验验证了方法的有效性。源代码已开源：https://github.com/Yushu-Li/ECALP。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have revolutionized machine learning by leveraging large pre-trained models to tackle various downstream tasks. Although label, training, and data efficiency have improved, many state-of-the-art VLMs still require task-specific hyperparameter tuning and fail to fully exploit test samples. To overcome these challenges, we propose a graph-based approach for label-efficient adaptation and inference. Our method dynamically constructs a graph over text prompts, few-shot examples, and test samples, using label propagation for inference without task-specific tuning. Unlike existing zero-shot label propagation techniques, our approach requires no additional unlabeled support set and effectively leverages the test sample manifold through dynamic graph expansion. We further introduce a context-aware feature re-weighting mechanism to improve task adaptation accuracy. Additionally, our method supports efficient graph expansion, enabling real-time inductive inference. Extensive evaluations on downstream tasks, such as fine-grained categorization and out-of-distribution generalization, demonstrate the effectiveness of our approach. The source code is available at https://github.com/Yushu-Li/ECALP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。