arXiv:2601.15249cs.LGcs.AI2026-01

用排序机制让作者真实评价论文,提升顶会最佳论文评选公正性

Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism

  • 让作者对投稿进行排序,用等序机制校正评审分数
  • 模拟实验显示新方法显著提高获奖论文质量
  • 即使只允许提名一篇,也能保证作者说真话,适用性更强

机器学习与人工智能顶会如NeurIPS和ICML每年收到数万篇投稿,评审质量与一致性面临挑战,尤其是最佳论文奖的评选近年来争议不断。本文提出一种作者辅助机制,通过等序机制收集作者对自己论文的排序评估,并据此调整原始评审分数,以更准确估计论文的真实质量。我们证明,当作者效用为调整后分数的凸加性函数时,其有动机如实报告;基于ICLR(2019–2023)和NeurIPS(2021–2023)公开评审数据验证了该凸性假设。特别地,在作者仅可提名一篇论文的场景下,即使效用函数仅为非递减加性,诚实报告仍为最优策略,显著放宽了先前研究的假设条件。针对作者重叠的常见情况,我们扩展了机制设计。模拟结果表明,该方法能显著提升最佳论文的评选质量。

原文摘要 · Abstract (English)

Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process. This challenge is particularly acute for best paper awards, which are an important part of the peer review process, yet whose selection has increasingly become a subject of debate in recent years. In this paper, we introduce an author-assisted mechanism to facilitate the selection of best paper awards. Our method employs the Isotonic Mechanism for eliciting authors' assessments of their own submissions in the form of a ranking, which is subsequently utilized to adjust the raw review scores for optimal estimation of the submissions' ground-truth quality. We demonstrate that authors are incentivized to report truthfully when their utility is a convex additive function of the adjusted scores, and we validate this convexity assumption for best paper awards using publicly accessible review data of ICLR from 2019 to 2023 and NeurIPS from 2021 to 2023. Crucially, in the special case where an author has a single quota -- that is, may nominate only one paper -- we prove that truthfulness holds even when the utility function is merely nondecreasing and additive. This finding represents a substantial relaxation of the assumptions required in prior work. For practical implementation, we extend our mechanism to accommodate the common scenario of overlapping authorship. Finally, simulation results demonstrate that our mechanism significantly improves the quality of papers selected for awards.

论文评审机制设计顶会奖项

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。