AI竞赛是评估生成式AI的黄金标准，能有效防作弊、保公平。
Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation
- 借鉴竞赛机制，建立抗泄露的评估体系
- 解决生成模型输入输出无限、无真值等核心难题
- 适合关注评估严谨性的研究者和评测平台
本文指出，当前生成式AI的实证评估正面临危机：传统机器学习评估方法难以应对现代生成模型的特性——输入输出空间近乎无限、缺乏明确真实答案，且输出高度依赖上下文反馈。更关键的是，数据泄露与污染问题尤为严重且难解。而人工智能竞赛领域已发展出有效措施应对作弊与泄露，具备极高的评估严谨性。因此，应将AI竞赛视为生成式AI评估的黄金标准，充分借鉴其方法与成果。
原文摘要 · Abstract (English)
In this position paper, we observe that empirical evaluation in Generative AI is at a crisis point since traditional ML evaluation and benchmarking strategies are insufficient to meet the needs of evaluating modern GenAI models and systems. There are many reasons for this, including the fact that these models typically have nearly unbounded input and output spaces, typically do not have a well defined ground truth target, and typically exhibit strong feedback loops and prediction dependence based on context of previous model outputs. On top of these critical issues, we argue that the problems of leakage and contamination are in fact the most important and difficult issues to address for GenAI evaluations. Interestingly, the field of AI Competitions has developed effective measures and practices to combat leakage for the purpose of counteracting cheating by bad actors within a competition setting. This makes AI Competitions an especially valuable (but underutilized) resource. Now is time for the field to view AI Competitions as the gold standard for empirical rigor in GenAI evaluation, and to harness and harvest their results with according value.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。