提醒研究者注意预处理参数也需调优,否则结果可能夸大。
Beyond algorithm hyperparameters: on preprocessing hyperparameters and associated pitfalls in machine learning applications
- 预处理步骤(如缺失值处理)也有可调参数,影响模型性能。
- 忽略预处理调优会导致模型评估结果被高估。
- 适合刚接触机器学习的科研人员参考,避免常见陷阱。
在监督式机器学习中,构建和评估预测模型常具挑战性,尤其对应用领域中的新手而言。通常关注算法超参数(如树模型中叶节点最小样本数)的调优,但数据预处理步骤(如缺失值处理方式)同样存在超参数。研究者在尝试不同预处理方法时,可能未意识到这已构成一种非正式、无系统的超参数调优,从而未报告或未考虑此优化过程。本文通过回顾与实证分析,揭示了模型生成与评估中算法与预处理超参数的不同处理方式,强调了由此产生的潜在问题,特别是可能导致性能声称过度乐观。旨在提升机器学习应用中预测建模的质量。
原文摘要 · Abstract (English)
Adequately generating and evaluating prediction models based on supervised machine learning (ML) is often challenging, especially for less experienced users in applied research areas. Special attention is required in settings where the model generation process involves hyperparameter tuning, i.e. data-driven optimization of different types of hyperparameters to improve the predictive performance of the resulting model. Discussions about tuning typically focus on the hyperparameters of the ML algorithm (e.g., the minimum number of observations in each terminal node for a tree-based algorithm). In this context, it is often neglected that hyperparameters also exist for the preprocessing steps that are applied to the data before it is provided to the algorithm (e.g., how to handle missing feature values in the data). As a consequence, users experimenting with different preprocessing options to improve model performance may be unaware that this constitutes a form of hyperparameter tuning, albeit informal and unsystematic, and thus may fail to report or account for this optimization. To illuminate this issue, this paper reviews and empirically illustrates different procedures for generating and evaluating prediction models, explicitly addressing the different ways algorithm and preprocessing hyperparameters are typically handled by applied ML users. By highlighting potential pitfalls, especially those that may lead to exaggerated performance claims, this review aims to further improve the quality of predictive modeling in ML applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。