发现并验证了近域外数据检测中的基准泄露问题,提出可复现的泄漏指纹诊断方法。
Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks
- 构建泄漏指纹:有监督解码性能接近1,无监督检测低于0.65
- 在52个实验设置中实现18/20敏感度与31/32特异性
- 适用于检测标准近域外基准是否被污染,适合评估者使用
在审计基于扰动的域外检测器时,发现其在文档基准上的AUROC仅为0.326,远低于0.5随机水平。原因是基准泄露:指定的“域外”类别实际上在训练中出现过,导致样本落入分布内拟合集,检测器反而因正确识别而受罚。删除该类别并重新训练35个模型后,得分提升至0.911。我们提炼出泄漏指纹——近乎完美的有监督解码能力(AUROC≈1)与低于0.65的无监督检测能力,并在52组设置(20个泄露、32个清洁)中验证,使用ResNet-50和ViT-B/16在CIFAR-10/100上实现了18/20敏感度和31/32特异性;匹配的拟合集排除控制达到20/20完美表现。对24个标准近/远域外基准对的实地审计仅在困难的CIFAR-100 vs CIFAR-10对触发,且无远域外对触发,证实特异性及跨数据集构建的清洁性。修正协议下,扰动信号可解码但不可检测:有监督读取器恢复信号(AUROC 0.87–1.00),而无监督检测器无法做到,且扰动方法未优于普通马氏距离。提供理论解释并撤回先前循环相关结论。贡献在于修正协议与验证泄漏诊断,而非新域外方法。
原文摘要 · Abstract (English)
While auditing a perturbation-based OOD detector on a document benchmark, we recorded an AUROC of 0.326 -- well below the 0.5 chance level. The cause is a benchmark leak: the designated "OOD" class is one the model was trained on, so its examples sit inside the in-distribution fit set and the detector is penalized for correctly ranking them as familiar. Deleting the class and retraining 35 models across two domains raises the score to 0.911. We distill the contamination into a leak fingerprint -- near-perfect supervised decodability (AUROC approximately 1) coupled with unsupervised detection collapsed below 0.65 -- and validate it on a controlled battery of 52 settings (20 leaked, 32 clean) across ResNet-50 and ViT-B/16 on CIFAR-10/100, achieving sensitivity 18/20 and specificity 31/32 in embedding space; the matched fit-set-exclusion controls are perfect at 20/20. An in-the-wild audit of 24 standard near/far OOD benchmark pairs fires on exactly one (the intrinsically hard CIFAR-100 vs CIFAR-10 pair) and on no far-OOD pair, confirming specificity and that standard cross-dataset construction is clean. Under the corrected protocol, perturbation signals are decodable but not detectable: a supervised reader recovers the OOD signal (AUROC 0.87-1.00) while no unsupervised detector does, and the perturbation method does not improve on plain Mahalanobis distance. We provide a theoretical account of why and, for transparency, retract an earlier circular correlation. The contributions are a corrected protocol and a validated leak diagnostic, not a new OOD method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。