研究指出，每项数据至少需10个评分才能可靠检验模型性能。
How Many Ratings per Item are Necessary for Reliable Significance Testing?
- 基于统计检验原理，推导出每项数据所需最少评分数。
- 发现5-10个评分不足以保证常见指标的可靠性。
- 适用于评估或设计可信的AI评测数据集的研究者。
机器学习评估的核心假设是模型与人类判断足够可靠，能通过准确率、精确率等单一标准进行评价。生成式AI的兴起打破了这一假设，因随机推理导致结果不稳定。然而，当前多数评估仍仅依赖每项输入少数几条输出。本文采用已有方法，评估现有或计划中的数据集是否具备足够的响应数以支持可靠的零假设显著性检验。结果显示，对于多数常用指标，每项仅收集5-10条响应（来自模型或人类评估团队）仍不足。我们对少数具备多拆分响应的现有金标准测试集进行了应用分析，发现这些数据集也缺乏足够响应数。本方法可帮助研究人员更科学地决定如何收集用于评估的AI数据。
原文摘要 · Abstract (English)
A cornerstone of machine learning evaluation is the (often hidden) assumption that model and human responses are reliable enough to evaluate models against unitary, authoritative, ``gold standard'' data, via simple metrics such as accuracy, precision, and recall. The generative AI revolution would seem to explode this assumption, given the critical role stochastic inference plays. Yet, in spite of public demand for more transparency in AI -- along with strong evidence that humans are unreliable judges -- estimates of model reliability are conventionally based on, at most, a few output responses per input item. We adapt a method, previously used to evaluate the reliability of various metrics and estimators for machine learning evaluation, to determine whether an (existing or planned) dataset has enough responses per item to assure reliable null hypothesis statistical testing. We show that, for many common metrics, collecting even 5-10 responses per item (from each model and team of human evaluators) is not sufficient. We apply our methods to several of the very few extant gold standard test sets with multiple disaggregated responses per item and show that even these datasets lack enough responses per item. We show how our methods can help AI researchers make better decisions about how to collect data for AI evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。