提出新方法揭示冻结编码器的真实性能,发现多数模型实际有效但表现平庸。
CANDOR: Chance-Calibrated Discordance in Frozen Foundation Encoders

- 设计对称分组的不一致度量CANDOR,确保随机猜测水平恒为0.5
- 22个编码器在60万+图像上测试,多数模型真实性能低于随机,但无完全无效
- 可提前评估编码器能力,适合医疗图像等需谨慎选择预训练模型的场景
冻结编码器的表现常被误判,因最近邻不一致度量受类别分布影响。本文提出CANDOR,通过等大小对称分组使机会水平恒定为0.5。在22个编码器、7个领域、20个数据集共605,443张图像上测试,修正后几乎所有模型性能均低于随机水平,但无完全失效。最佳肺部模型在气胸识别上仅达84.5 AUROC,仍有18.4%阳性病例比同类型更接近反类图像;同一模型在鸟类分类上达4.5,但在胸部和青光眼任务上分别仅42.8和49.8,处于随机水平甚至更差。这限制了任何Lipschitz头的归一化间隔,但11种头中一种可正确解决除2.8%外的所有错误,且单个头错35.9%案例。缺陷源于选择而非信息缺失。擦除保留与崩溃相关,但与目标、规模、近期性或病灶大小无关。因机会水平固定,CANDOR可在训练头部前使用,识别编码器支持弱的发现。
原文摘要 · Abstract (English)
Frozen encoders are chosen by how well a lightweight head reads a finding from their features, not whether the geometry separates it. Nearest-neighbor discordance does, but with unequal banks the opposite-label neighbor wins on density, not geometry, so prevalence alone makes an uninformed encoder look blind. We introduce CANDOR, a discordance measure whose equal-size banks are symmetric under a label swap, fixing its chance level at exactly one half. Across 22 encoders, 20 datasets from 7 domains, and 605,443 images, this correction reverses the conclusion. Collapse falls below chance almost everywhere, so no encoder is blind, yet all are weak: the best chest model reads pneumothorax at 84.5 AUROC and still places 18.4% of those positives nearer an opposite-label film than its own kind in the same hospital. The same encoder that resolves bird species at 4.5 leaves chest findings at 42.8 and glaucoma at 49.8, at chance and worse than random weights. Such a case caps the normalized margin of any Lipschitz head, yet some head among eleven is correct on all but 2.8% of cases where one head misses 35.9%: the deficit is selection, not information. Erasure retention is associated with collapse; we detect no association with the objective, scale, recency, or size of the finding. Because the chance level is fixed, CANDOR can be read before any head is trained, flagging which findings a frozen encoder supports poorly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。