通过拓扑对齐提升CLIP在少样本学习中的表现
Topology-Aware CLIP Few-Shot Learning
- 用拓扑差异损失对齐视觉与文本表征的结构
- 仅优化轻量级参数，平均准确率提升1-2%
- 适合需要高效适配大模型的少样本任务
将大型视觉语言模型（如CLIP）高效适配到少样本学习面临保持预训练知识与任务特化之间的平衡难题。现有方法常忽略模型隐空间中的重要结构信息。本文提出一种拓扑感知微调方法，将表示拓扑差异（RTD）融入任务残差（TR）框架。通过联合使用RTD与交叉熵损失，显式对齐视觉与文本表征的拓扑结构，同时冻结基础VLM编码器，仅优化轻量级任务残差参数，有效利用拓扑信息。在6个不同基准数据集上，该方法显著提升少样本性能，平均准确率相较基线方法提高1-2%。本工作为增强VLM少样本能力提供了有效策略。
原文摘要 · Abstract (English)
Efficiently adapting large Vision-Language Models (VLMs) like CLIP for few-shot learning poses challenges in balancing pre-trained knowledge retention and task-specific adaptation. Existing methods often overlook valuable structural information within the VLM's latent space. We introduce a topology-aware tuning approach integrating Representation Topology Divergence (RTD) into the Task Residual (TR) framework. By explicitly aligning the topological structures of visual and text representations using a combined RTD and Cross-Entropy loss, while freezing base VLM encoders, our method enhances few-shot performance. We optimize only lightweight Task Residual parameters, effectively leveraging topological information. Across 6 diverse benchmark datasets, our approach demonstrates significant gains, achieving an average accuracy improvement of 1-2\% over relevant baseline methods in few-shot settings. This work presents an effective strategy to boost VLM few-shot capabilities by incorporating topological alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。