统一机器学习评估指标,解决多框架结果不可比问题
AllMetrics: A Unified Python Library for Standardized Metric Evaluation and Robust Data Validation in Machine Learning
- 构建统一的Python库,规范回归、分类等任务的指标计算
- 通过输入验证和模块化API,提升评估结果可复现性
- 支持医疗、金融等多领域数据,适合需要可靠评估的研究者
机器学习模型依赖一致准确的性能指标进行评估与比较。然而,现有库普遍存在碎片化、实现不一致和数据验证不足的问题,导致结果不可靠。各库独立开发,缺乏统一标准,尤其在支持的任务类型上差异明显,各自采用不同的指标计算、输入输出格式与错误处理方式,造成实现差异(ID)和报告差异(RD),难以跨框架比较或确保评估可靠性。为此,我们提出AllMetrics,一个开源的统一Python库,旨在标准化回归、分类、聚类、分割及图像到图像转换等多种任务的指标评估。通过可配置参数实现多分类任务的类别特异性报告,并引入任务特定参数解决不同实现间的计算偏差。我们在医疗、金融、房地产等领域的多个数据集上测试该库,并与Python、Matlab、R组件对比,验证其结果一致性。AllMetrics结合模块化API与强大的输入验证机制,确保评估过程的可复现性与可靠性。本文阐述其设计原则、架构组件及实证分析,证明其能有效减少评估误差,增强机器学习工作流的可信度。
原文摘要 · Abstract (English)
Machine learning (ML) models rely heavily on consistent and accurate performance metrics to evaluate and compare their effectiveness. However, existing libraries often suffer from fragmentation, inconsistent implementations, and insufficient data validation protocols, leading to unreliable results. Existing libraries have often been developed independently and without adherence to a unified standard, particularly concerning the specific tasks they aim to support. As a result, each library tends to adopt its conventions for metric computation, input/output formatting, error handling, and data validation protocols. This lack of standardization leads to both implementation differences (ID) and reporting differences (RD), making it difficult to compare results across frameworks or ensure reliable evaluations. To address these issues, we introduce AllMetrics, an open-source unified Python library designed to standardize metric evaluation across diverse ML tasks, including regression, classification, clustering, segmentation, and image-to-image translation. The library implements class-specific reporting for multi-class tasks through configurable parameters to cover all use cases, while incorporating task-specific parameters to resolve metric computation discrepancies across implementations. Various datasets from domains like healthcare, finance, and real estate were applied to our library and compared with Python, Matlab, and R components to identify which yield similar results. AllMetrics combines a modular Application Programming Interface (API) with robust input validation mechanisms to ensure reproducibility and reliability in model evaluation. This paper presents the design principles, architectural components, and empirical analyses demonstrating the ability to mitigate evaluation errors and to enhance the trustworthiness of ML workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。