arXiv:2607.28967cs.CVcs.LG2026-07

用视觉原型提升提示调优效率,无需标签和训练

Visual Distribution Anchoring for Efficient Prompt Tuning

论文配图:Visual Distribution Anchoring for Efficient Prompt Tuning
图 1 · 摘自论文原文
  • 通过无监督聚类构建类别视觉原型,替代传统提示调优
  • 在10个图像迁移任务中平均提升3.35点,最高提升3.22点
  • 适合零样本迁移、模型蒸馏等场景,无需目标数据标注

提示调优虽以少量可训练参数适配视觉-语言模型,但现有方法在效率与适应性间存在权衡:静态文本提示易过拟合源类别,图像条件提示增加实例计算开销,多模态调优则修改视觉分支。本文提出VDA(视觉分布锚定)——一种无需训练的目标域适应框架,通过离线估计未标注目标域的类别级视觉原型,增强冻结的语义分类器。研究发现仅凭类别名无法重建有效原型,因名称表征语义而非目标域外观;通过冻结的语义分类器与领域模板分类器将目标图像划分为相关组别,置信度排序的特征形成归一化原型,并以单一全局权重融合至分类器。该方法无需目标标签、目标端优化、均匀先验假设、迭代精炼或测试查询访问,生成固定可缓存分类器。控制实验表明,类别特定划分带来性能提升,且视觉局部伪标签错误仍具价值。在十次ImageNet到目标域迁移中,相同冻结设计使零样本CLIP、TCP和MaPLe分别提升3.22、3.39、3.35点,九个目标在所有设置中均获益。其视觉修正进一步使无泄漏提示知识蒸馏(PromptKD)提升2.79点,兼容零样本、源提示、多模态提示及目标蒸馏分类器。

原文摘要 · Abstract (English)

Prompt tuning adapts vision--language models with few trainable parameters, but existing approaches trade off efficiency and adaptation: static textual prompts can overfit source classes, image-conditioned prompts add per-instance computation, and multimodal tuning modifies the visual branch. We propose VDA (Visual Distribution Anchoring), a training-free target adaptation framework that augments a frozen semantic classifier with class-level visual prototypes estimated offline from an unlabeled target pool. We first ask whether prototypes can be synthesized from class names. A text-to-centroid mapper reconstructs held-out source prototypes but fails under dataset shift because class names specify semantic identity, not target-domain appearance. An oracle analysis confirms that true target prototypes are highly discriminative. VDA therefore uses frozen semantic and domain-template classifiers to partition unlabeled target images into class-correlated groups. Confidence-ranked image features form normalized prototypes, fused with the semantic classifier using one global weight. Adaptation requires no target labels, target-side optimization, uniform class-prior assumption, iterative refinement, or test-query access, and yields a fixed, cacheable classifier. Controlled experiments show that class-specific partitioning drives gains and that visually local pseudo-label errors can remain useful despite being class-incorrect. Across ten ImageNet-to-target transfers, the same frozen design improves zero-shot CLIP, TCP, and MaPLe by 3.22, 3.39, and 3.35 points, respectively, improving nine of ten targets in every setting. Its visual correction further improves leakage-free PromptKD by 2.79 points, complementing zero-shot, source-prompted, multimodal-prompted, and target-distilled classifiers.

提示调优零样本迁移视觉原型无监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。