arXiv:2508.04489cs.LGcs.AI2025-08ICML被引 2

给分类错误设计分级评分,更精准评估模型失误影响。

Hierarchical Scoring for Machine Learning Classifier Error Impact Evaluation

  • 用评分树表示类别层级关系,按距离远近给错误打分
  • 相比传统全对全错,能区分错误严重程度并量化影响
  • 适合关注误判后果的场景,如医疗、安防等高风险领域

机器学习模型常用于预测样本类别,目标检测则是分类的扩展,包含通过边界框定位目标。传统评估方法仅当预测标签与真实标签不同时视为错误,即“通过/失败”评分,所有误判被视为同等。但在许多场景中,类别可组织成具有层级结构的分类体系,反映数据间关系或人为对误判的重视程度。此时,采用层级评分机制可依据预测与真实标签在评分树中的距离,返回模型表现,给予部分得分而非简单对错,实现对误判影响的更细粒度分析。本文提出多种复杂度不同的层级评分指标,利用评分树编码类别间关系,生成反映树中距离的评价指标。在抽象案例中,使用代表三种权重策略的评分树进行演示,并评估其抑制特定类型错误的效果。结果表明,这些指标能更精细捕捉错误特征,且评分树支持灵活调整。该工作展示了一种新评估范式:不仅看错误数量,更关注错误类型与实际影响。论文代码将在发表时开源。

原文摘要 · Abstract (English)

A common use of machine learning (ML) models is predicting the class of a sample. Object detection is an extension of classification that includes localization of the object via a bounding box within the sample. Classification, and by extension object detection, is typically evaluated by counting a prediction as incorrect if the predicted label does not match the ground truth label. This pass/fail scoring treats all misclassifications as equivalent. In many cases, class labels can be organized into a class taxonomy with a hierarchical structure to either reflect relationships among the data or operator valuation of misclassifications. When such a hierarchical structure exists, hierarchical scoring metrics can return the model performance of a given prediction related to the distance between the prediction and the ground truth label. Such metrics can be viewed as giving partial credit to predictions instead of pass/fail, enabling a finer-grained understanding of the impact of misclassifications. This work develops hierarchical scoring metrics varying in complexity that utilize scoring trees to encode relationships between class labels and produce metrics that reflect distance in the scoring tree. The scoring metrics are demonstrated on an abstract use case with scoring trees that represent three weighting strategies and evaluated by the kind of errors discouraged. Results demonstrate that these metrics capture errors with finer granularity and the scoring trees enable tuning. This work demonstrates an approach to evaluating ML performance that ranks models not only by how many errors are made but by the kind or impact of errors. Python implementations of the scoring metrics will be available in an open-source repository at time of publication.

模型评估分类误差层级评分

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。