首个跨范式深度伪造图像检测基准，统一评测商业API、大模型与开源工具。
VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection
- 构建统一100张对抗样本集，涵盖8类真实场景边缘案例。
- 商业API表现最佳，但部分模型默认阈值存在误判风险。
- 仅用准确率或F1不可靠，需结合MCC与特异性综合评估。
深度伪造图像检测依赖三种不同范式——商业API、零样本视觉语言模型（LLMs）和开源检测器——但缺乏统一评估协议，难以直接比较。我们提出VendorBench-100，一个跨范式基准，使用单一对抗性100图像语料库、统一输出格式和共同评估框架，评测36个代表性模型。模型主要按马修斯相关系数（MCC）排名，同时报告ROC-AUC作为无阈值的排序能力度量。该基准强调真实世界难度，包含八类边缘案例：人脸替换、文本生成静帧、AI照片编辑、虚拟角色合成、来源不透明图像及压缩研究帧。商业API在中位数性能上领先，其次为视觉大模型，开源检测器虽整体较弱，但个别模型可媲美最优大模型。所有36个模型中，MCC与ROC-AUC强相关（皮尔逊相关系数r ~ 0.86），但更关键发现是部分表现优异模型在默认阈值下存在校准偏差，导致高ROC-AUC可能夸大实际部署可行性。此外，在类别不平衡的测试集中，原始准确率与F1不可靠：一味预测“伪造”者得分虚高，却无真实判别能力。单一指标均不可靠：必须搭配MCC与特异性共同分析ROC-AUC与准确率。完整评估框架与结果已公开。代码与数据：https://github.com/sharayu-20/vendorbench-100
原文摘要 · Abstract (English)
Deepfake image detection is served by three fundamentally different paradigms - commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors - that are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. Models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing size, it emphasizes real-world difficulty through a taxonomy of eight edge-case families such as face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors, though individual open-source models remain competitive with the best LLMs. Across all 36 models, MCC and ROC-AUC are strongly correlated (Pearson r ~ 0.86); the more consequential finding is narrower and one-directional: a subset of otherwise strong rankers are miscalibrated at their shipped default threshold, so a high ROC-AUC can overstate real-world deployability. Separately, raw accuracy and F1 are unreliable on this corpus's imbalanced class split, since a model that predicts "fake" indiscriminately scores deceptively well on both while offering no real discriminative skill. No single metric is safe in isolation: MCC and specificity should always accompany ROC-AUC and accuracy. We release the complete evaluation framework and results. Code and data: https://github.com/sharayu-20/vendorbench-100
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。