发现视觉语言模型少样本适配中混合比例可免费近似最优，但性能瓶颈不在比例本身。
The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models
- 用理论最优比例替代传统调参，通过支持集直接估计无需验证数据
- 理论最优比例在950个测试单元中仍比理想值低8.5分，因文本图像距离的偏移被误判为偏差
- 无验证数据的线性探测器反而超越调优混合比，说明性能天花板在模型而非超参
许多视觉语言模型的少样本适配方法通过零样本文本原型与K个标注图像特征均值的凸组合进行分类，仅用一个混合比例在保留标签上调节，常在测试集上进行。我们追问该方法的偏差-方差合理性：最优比例是什么？能否不依赖验证数据估计？性能是否取决于它？首先，最小化原型均方误差的比率有闭式解，其支持集插值恰好是正部分James-Stein系数，趋向于文本原型。在4,800个测试单元（10个数据集、5个主干网络包括SigLIP、5个样本数、5个随机种子、4个提示层级）中，该理论最优比例虽可靠，却仍是错误量的估计：在950个主层级单元中，其表现比测试集最优比例低8.5分。该比例趋近1，即舍弃文本先验，采用最近类均值分类器，因为78%的文本-图像原型距离被误认为是偏差，而最大值分类器基本抵消了这部分影响。我们证明此机制，并通过反事实分析将损害占比上限定为26%。其次，仅使用支持集的留一法即可得到接近最优的比例，表明其可在无验证数据情况下估计。第三，无需验证的线性探测器表现优于甚至超过最优调参的混合比例：CLAP平均+1.9分，LP++平均+1.5分；当K≥4时，全部四个无验证基线均高于最优调参结果，且差异置信区间不含零。这些结果表明性能上限在于模型类别，而非超参数选择：比例可免费设至近优，但性能仍未达到极限。
原文摘要 · Abstract (English)
Many few-shot adaptation methods for vision-language models classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features, with a single blending ratio routinely tuned on held-out labels, often on the test set itself. We ask what the family's own bias-variance justification invites: what is the right ratio, can it be estimated without validation data, and is finding it where the performance is? First, the ratio minimising prototype mean-squared error has a closed form whose support-set plug-in is exactly a positive-part James-Stein coefficient shrinking towards the text prototype. Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity: on the 950 primary-tier cells where it is defined it trails a test-set-oracle ratio by 8.5 points. It saturates near 1, discarding the text prior for a nearest-class-mean classifier, because 78% of the text-image prototype distance it treats as bias is a class-independent offset that the arg max largely cancels. We prove the mechanism and bound its share of the damage at 26% by a counterfactual. Second, leave-one-out on the support set alone sets a ratio landing within 0.9 points of the oracle blend, so it is estimable without validation data. Third, validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average, and at K >= 4 all four validation-free baselines sit above the oracle, the linear probes by margins excluding zero. These results locate the ceiling in the model class, not the hyperparameter: the ratio can be set near-optimally for free, and it is still not where the performance is. Code, cached features, per-cell records: https://huggingface.co/datasets/Liangzhi-Li/clipbench-blending
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。