如何正确评估机器学习系统，避免结论误导。
Good practices for evaluation of machine learning systems
- 设计评估流程时需谨慎选择数据与指标
- 错误的评估会导致模型性能误判
- 适合研究者和工程师参考评估规范
机器学习实验中的诸多决策——如训练数据、特征、模型结构、超参数、测试数据等——都会影响最终结果。其中，评估流程的设计尤为关键，它决定了实验结论能否推广到未见数据，以及是否对实际应用有意义。若数据选取不当、评价指标错误或模型比较的显著性被高估，结论可能产生误导，导致次优开发决策。因此，评估协议应在实验开始前精心设计。本文讨论评估协议设计的核心要素：数据选择、指标选择和统计显著性。虽非详尽教程，但旨在阐明各环节应遵循的主要准则。文中以语音处理领域为例，并列出每类问题的常见错误，帮助研究者规避风险。
原文摘要 · Abstract (English)
Many development decisions affect the results obtained from ML experiments: training data, features, model architecture, hyperparameters, test data, etc. Among these aspects, arguably the most important design decisions are those that involve the evaluation procedure. This procedure is what determines whether the conclusions drawn from the experiments will or will not generalize to unseen data and whether they will be relevant to the application of interest. If the data is incorrectly selected, the wrong metric is chosen for evaluation or the significance of the comparisons between models is overestimated, conclusions may be misleading or result in suboptimal development decisions. To avoid such problems, the evaluation protocol should be very carefully designed before experimentation starts. In this work we discuss the main aspects involved in the design of the evaluation protocol: data selection, metric selection, and statistical significance. This document is not meant to be an exhaustive tutorial on each of these aspects. Instead, the goal is to explain the main guidelines that should be followed in each case. We include examples taken from the speech processing field, and provide a list of common mistakes related to each aspect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。