用长期影响力对齐LLM评审，提升发现高价值论文的能力
ReviewGuard: Aligning LLM-Assisted Peer Review with Long-Term Scientific Impact
- 用引文数据训练模型，让AI评审更关注未来影响力而非即时偏好
- 在拒稿后仍获高引的论文中，相关性达0.776，远超人类评审的0.492
- 可识别出5.6倍于人类的高潜力被拒论文，适合期刊编辑辅助决策
同行评审是科学质量控制的核心，但常低估那些后来产生重大引用影响的论文。尽管前沿大语言模型在自动化评审方面展现出潜力，但它们主要模仿人类评审的即时偏好，而非预测长期科学价值。我们提出ReviewGuard，一种两阶段框架，使LLM生成的评审与基于引文的长期科学影响力对齐，而非与同期评审判断一致。在包含20,861篇来自OpenReview的AI/ML论文并附带Semantic Scholar引文数据的基准上，ReviewGuard在被拒后仍获高引的论文上与未来引文数的斯皮尔曼相关系数达到0.776，优于人类评审（0.492）和监督式专家模型（0.681）。在相同决策阈值下，ReviewGuard能识别出10.2%的高影响力被拒论文，而人类仅能识别1.8%，提升5.6倍。结果表明，以影响力为导向的强化学习可为编辑提供互补信号，识别高潜力工作，而不替代人类判断。
原文摘要 · Abstract (English)
Peer review is central to scientific quality control, yet it can undervalue papers that later achieve substantial citation impact. While frontier large language models have shown promise in automating aspects of peer review, they primarily mimic human reviewer preferences rather than predict long-term scientific value. We introduce ReviewGuard, a two-stage framework that aligns LLM-generated reviews with citation-based estimates of long-term scientific impact rather than contemporaneous reviewer judgments. On 20,861 AI/ML papers from OpenReview augmented with Semantic Scholar citation data, ReviewGuard achieves a Spearman correlation of \r{ho} = 0.776 with future citations on rejected-then-published papers, outperforming human reviewers (\r{ho} = 0.492) and a supervised Expert model (\r{ho} = 0.681). Under the same decision threshold, ReviewGuard flags 10.2% of high-impact rejected papers, compared with 1.8% for human reviewers, corresponding to a 5.6x improvement. Our results demonstrate that impact-aligned reinforcement learning can provide editors with a complementary signal for identifying high-potential work, without replacing human judgment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。