揭示防御模型隐含的准确率损失,帮用户看清鲁棒性代价。
VanillaBench: The Hidden Accuracy Cost of Adversarial Robustness

- 构建系统化基准 VanillaBench,量化对抗训练模型与原始模型的准确率差距。
- 平均准确率下降7.7至29.5个百分点,最鲁棒模型仍比同期普通模型低4.0至21.0点。
- 在同架构对比下,准确率损失仍达3.5至17.5个百分点,凸显真实代价。
过去十年中,对抗鲁棒性研究已产出数百个防御模型,但文献普遍仅孤立报告鲁棒性结果:只展示鲁棒模型的标准(干净)准确率和对抗准确率,却极少量化其与对应原始模型的差距。我们提出 VanillaBench,一个系统性基准,使这一差距显式化。针对 RobustBench 中四个威胁模型下的所有 186 个鲁棒模型,我们计算其与 Papers with Code 中多个原始参照模型的准确率差异,涵盖全部条目与无额外数据条目、该鲁棒模型发表年份的最佳原始模型,以及架构匹配基线。结果显示,相对于最佳原始模型,鲁棒模型的平均干净准确率下降 -7.7 至 -29.5 个百分点;即使是最鲁棒的模型,在各赛道上仍落后于同期原始模型 4.0 至 21.0 个百分点。在架构匹配对比中,平均差距为 -3.5 至 -17.5 个百分点;若仅使用同一架构的原始准确率而非近似值,则差距缩小至 -4.0 至 -14.0 个百分点。这些结果表明,鲁棒性与准确率之间的权衡远大于论文通常传达的程度。此信息对实践者和决策者至关重要。在实际部署中,鲁棒性的准确率成本直接影响业务表现,但现有发表文献未提供评估该成本所需的原始基线。我们主张未来鲁棒性评估应将原始参照准确率差距作为标准组成部分。
原文摘要 · Abstract (English)
Adversarial robustness research has produced hundreds of defended models over the past decade, yet the literature almost universally reports robustness results in isolation: standard (clean) accuracy and adversarial accuracy of the robust model are shown, but the gap to the corresponding vanilla model is rarely quantified. We introduce VanillaBench, a systematic benchmark that makes this gap explicit. For every adversarially-trained model catalogued by RobustBench across four threat models, we compute the accuracy difference against multiple vanilla references from Papers with Code, computed over both all entries and no-extra-data entries, the best vanilla model as of the robust model's publication year, and an architecture-matched baseline. Across all 186 robust models, the mean delta clean relative to the best vanilla model ranges from -7.7 to -29.5 percentage points, and even the single most robust model per track still trails its temporal vanilla counterpart by 4.0-21.0 points. The architecture-matched comparison, which isolates the effect of adversarial training from architectural differences, reveals a mean gap of -3.5 to -17.5 points. Restricting this architecture-matched comparison to models whose vanilla accuracy is known for the exact same architecture, rather than approximated from a related one, narrows the gap to -4.0 to -14.0 points. These results demonstrate that the robustness-accuracy trade-off is substantially larger than what is typically conveyed by individual papers. This information is critical for practitioners and decision-makers. When deploying models in real-world settings, the accuracy cost of robustness directly affects business outcomes, yet current publications do not provide the vanilla baseline needed to assess it. We argue that future robustness evaluations should report vanilla-referenced accuracy gaps as a standard component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。