统一评估分类器的误判与分布外样本,提升实际部署可靠性。
From Misclassifications to Outliers: Joint Reliability Assessment in Classification
- 用双评分函数同时评估误判与分布外输入
- 在OpenOOD上可靠性显著优于传统单评分方法
- 适合需要高可信度的现实场景模型部署
构建可靠的分类器是机器学习在真实应用中部署的核心挑战。可靠系统不仅应检测分布外(OOD)输入,还应在分布内(ID)预测可能的错误,通过为潜在误分类样本分配低置信度来实现。然而,以往研究将OOD检测与故障预测视为独立问题,忽略了二者密切关联。本文主张需联合评估二者。为此,提出统一评估框架,引入新指标DS-F1和DS-AURC,其中DS代表双评分函数。在OpenOOD基准上的实验表明,双评分函数使分类器可靠性显著优于传统单评分方法。分析还显示,在简单或远距离分布偏移下,基于OOD的方法有明显优势,但在更具挑战性的近距离分布偏移下收益有限。除评估外,扩展了可靠分类器SURE,提出SURE+,在多种场景下显著提升可靠性。本工作建立了可信分类的新基准,并为实际部署提供实用指导。源码已公开于https://github.com/Intellindust-AI-Lab/SUREPlus。
原文摘要 · Abstract (English)
Building reliable classifiers is a fundamental challenge for deploying machine learning in real-world applications. A reliable system should not only detect out-of-distribution (OOD) inputs but also anticipate in-distribution (ID) errors by assigning low confidence to potentially misclassified samples. Yet, most prior work treats OOD detection and failure prediction as separated problems, overlooking their closed connection. We argue that reliability requires evaluating them jointly. To this end, we propose a unified evaluation framework that integrates OOD detection and failure prediction, quantified by our new metrics DS-F1 and DS-AURC, where DS denotes double scoring functions. Experiments on the OpenOOD benchmark show that double scoring functions yield classifiers that are substantially more reliable than traditional single scoring approaches. Our analysis further reveals that OOD-based approaches provide notable gains under simple or far-OOD shifts, but only marginal benefits under more challenging near-OOD conditions. Beyond evaluation, we extend the reliable classifier SURE and introduce SURE+, a new approach that significantly improves reliability across diverse scenarios. Together, our framework, metrics, and method establish a new benchmark for trustworthy classification and offer practical guidance for deploying robust models in real-world settings. The source code is publicly available at https://github.com/Intellindust-AI-Lab/SUREPlus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。