arXiv:2411.12032cs.LGcs.SE2024-11被引 4

不同编程语言的机器学习评估指标差异大,亟需统一标准。

Machine Learning Evaluation Metric Discrepancies across Programming Languages and Their Components: Need for Standardization

  • 对比了Python、R、Matlab在多种任务中的评估指标
  • 仅部分指标(如准确率、AUC、MAE)跨平台一致
  • 建议统一指标以提升模型可比性和复现性

本研究评估了分类、回归、聚类、相关分析、统计检验、分割和图像到图像翻译等任务的评估指标,对比了Python库、R包和Matlab函数之间的表现。结果显示,仅部分指标在不同平台间保持一致,如二分类中的准确率、平衡准确率、Cohen's Kappa、F-beta分数、MCC、几何均值、AUC和对数损失;多分类中的准确率、Cohen's Kappa和F-beta分数;回归中的MAE、MSE、RMSE、MAPE、解释方差、中位绝对误差、MSLE和Huber损失;聚类中的Davies-Bouldin指数和Calinski-Harabasz指数;相关分析中的皮尔逊、斯皮尔曼、肯德尔tau、互信息、距离相关性、Percbend、Shepherd和偏相关;统计检验中的配对t检验、卡方检验、ANOVA、Kruskal-Wallis检验、Shapiro-Wilk检验、Welch t检验和Bartlett检验;2D分割中的准确率、精确率和召回率;3D分割中的准确率;2D-I2I翻译中的MAE、MSE、RMSE和R平方;以及3D-I2I翻译中的MAE、MSE和RMSE。然而,在精确率、召回率、F1分数、聚类的WCSS、多种统计检验及分割的IoU等指标上存在显著差异。研究结论指出,机器学习评估指标亟需标准化,并建议未来研究采用一致指标以实现有效比较。

原文摘要 · Abstract (English)

This study evaluates metrics for tasks such as classification, regression, clustering, correlation analysis, statistical tests, segmentation, and image-to-image (I2I) translation. Metrics were compared across Python libraries, R packages, and Matlab functions to assess their consistency and highlight discrepancies. The findings underscore the need for a unified roadmap to standardize metrics, ensuring reliable and reproducible ML evaluations across platforms. This study examined a wide range of evaluation metrics across various tasks and found only some to be consistent across platforms, such as (i) Accuracy, Balanced Accuracy, Cohens Kappa, F-beta Score, MCC, Geometric Mean, AUC, and Log Loss in binary classification; (ii) Accuracy, Cohens Kappa, and F-beta Score in multi-class classification; (iii) MAE, MSE, RMSE, MAPE, Explained Variance, Median AE, MSLE, and Huber in regression; (iv) Davies-Bouldin Index and Calinski-Harabasz Index in clustering; (v) Pearson, Spearman, Kendall's Tau, Mutual Information, Distance Correlation, Percbend, Shepherd, and Partial Correlation in correlation analysis; (vi) Paired t-test, Chi-Square Test, ANOVA, Kruskal-Wallis Test, Shapiro-Wilk Test, Welchs t-test, and Bartlett's test in statistical tests; (vii) Accuracy, Precision, and Recall in 2D segmentation; (viii) Accuracy in 3D segmentation; (ix) MAE, MSE, RMSE, and R-Squared in 2D-I2I translation; and (x) MAE, MSE, and RMSE in 3D-I2I translation. Given observation of discrepancies in a number of metrics (e.g. precision, recall and F1 score in binary classification, WCSS in clustering, multiple statistical tests, and IoU in segmentation, amongst multiple metrics), this study concludes that ML evaluation metrics require standardization and recommends that future research use consistent metrics for different tasks to effectively compare ML techniques and solutions.

评估指标标准化跨平台机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。