作者自评论文质量能有效预测未来影响力,比同行评审更准。
How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
- 让作者对自己的多篇投稿按质量自评,利用其独特认知判断潜力。
- 自评最高论文的引用量是最低的两倍,尤其能发现超150次引用的高影响力作品。
- 适合关注科研选题、投稿策略或评估研究价值的学者参考。
学术同行评审不仅确保内容正确性,还试图识别具有高科学潜力、能引领未来研究方向的工作。在人工智能等快速发展的领域,这一任务尤为关键,但随着投稿量激增,难度日益加大。本文探讨一种未被充分重视的指标:作者对自己同一次会议多个投稿的自评排名。基于博弈论推理,我们假设自评具有信息量,因作者对自身工作的概念深度和长期前景有独特理解。我们在顶级AI会议上开展大规模实验,1342名研究人员对其2592篇投稿进行了质量自评。跟踪超过一年的结果发现,作者自评最高的论文引用量是自评最低论文的两倍;自评在识别高引用论文(超150次引用)方面尤为有效。此外,自评分数比同行评审评分更能预测未来引用数。该结论在控制预印本发布时间和自我引用等混杂因素后依然稳健。结果表明,作者自评可作为同行评审的可靠补充,有效识别并提升人工智能领域的高影响力研究。
原文摘要 · Abstract (English)
Peer review in academic research aims not only to ensure factual correctness but also to identify work of high scientific potential that can shape future research directions. This task is especially critical in fast-moving fields such as artificial intelligence (AI), yet it has become increasingly difficult given the rapid growth of submissions. In this paper, we investigate an underexplored measure for identifying high-impact research: authors' own rankings of their multiple submissions to the same AI conference. Grounded in game-theoretic reasoning, we hypothesize that self-rankings are informative because authors possess unique understanding of their work's conceptual depth and long-term promise. To test this hypothesis, we conducted a large-scale experiment at a leading AI conference, where 1,342 researchers self-ranked their 2,592 submissions by perceived quality. Tracking outcomes over more than a year, we found that papers ranked highest by their authors received twice as many citations as their lowest-ranked counterparts; self-rankings were especially effective at identifying highly cited papers (those with over 150 citations). Moreover, we showed that self-rankings outperformed peer review scores in predicting future citation counts. Our results remained robust after accounting for confounders such as preprint posting time and self-citations. Together, these findings demonstrate that authors' self-rankings provide a reliable and valuable complement to peer review for identifying and elevating high-impact research in AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。