arXiv:2501.06492cs.LG2025-01被引 12

提出灵活数据划分算法,帮机器学习选最优验证方法

A New Flexible Train-Test Split Algorithm, an approach for choosing among the Hold-out, K-fold cross-validation, and Hold-out iteration

  • 构建可系统评估多种验证策略的Python框架
  • 实验显示无统一最优验证方法,效果依赖算法与数据特性
  • 适合关注模型泛化性能的研究者和实践者

选择合适的训练-测试划分策略是机器学习中的关键步骤,但验证方法常基于默认或惯例设置,未充分考虑其对泛化能力与真实性能的影响。本文提出一种灵活的Python框架,系统评估七种常用机器学习算法(决策树、K近邻、朴素贝叶斯、逻辑回归、校准线性支持向量机、直方图梯度提升)在不同验证策略下的表现,包括10%~90%的留出法、k值为3~15的k折交叉验证、重复留出法及嵌套交叉验证。在三个不同规模的生物医学数据集上,使用ROC-AUC、准确率和马修斯相关系数进行评估。结果表明,没有单一验证策略在所有算法和数据集上始终最优,最优选择取决于算法、数据特征与评估指标的交互作用。

原文摘要 · Abstract (English)

Choosing an appropriate strategy for partitioning data into training and evaluation sets is a critical step in machine learning, yet validation methods are often selected using default or conventional settings without considering their impact on generalizability and real-world performance. Common approaches such as hold-out validation or k-fold cross-validation with fixed k values are frequently applied based solely on empirical practice. To address this issue, we propose a flexible Python-based framework that systematically examines how different validation strategies affect predictive performance across seven widely used machine learning algorithms, including Decision Trees, K-Nearest Neighbors, Naive Bayes variants, Logistic Regression, calibrated linear Support Vector Machines, and histogram-based gradient boosting. The framework evaluates these methods under a wide range of validation schemes, including hold-out splits from 10% to 90%, k-fold cross-validation with k between 3 and 15, repeated hold-out, and nested cross-validation. The framework is applied to three biomedical datasets of varying size, and performance is assessed using ROC-AUC, accuracy, and the Matthews correlation coefficient. The results show that no single validation strategy consistently outperforms others across all algorithms and datasets, indicating that optimal validation depends on the interaction between the algorithm, dataset characteristics, and evaluation metric.

验证策略机器学习数据划分

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。