用AI给审稿人提反馈，能显著提升审稿质量与作者互动。
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
- 通过多LLM自动检测审稿中的模糊、误解和不专业问题并给出改进建议。
- 27%收到反馈的审稿人修改了评论，超1.2万条建议被采纳，平均增80词。
- 适合关注提升学术评审效率与质量的研究者与会议组织者。
AI顶会审稿因投稿量激增而压力倍增，导致评审质量下降和作者不满。为此，我们开发了Review Feedback Agent，利用多个大语言模型（LLMs）对审稿意见中的模糊表述、内容误解和不专业言辞提供自动化反馈，以提升评审的清晰度与可操作性。该系统在ICLR 2025中以大规模随机对照试验形式部署，向超过20,000份随机选取的审稿意见提供可选反馈。为确保反馈质量，我们还构建了一套由LLM驱动的自动化可靠性测试体系，仅当反馈通过全部测试时才发送。结果显示，27%的接收反馈者修改了原审稿意见，超过12,000条反馈建议被采纳；采纳后审稿平均增加80词，盲评研究者评价其信息量显著提升。同时，接受反馈的审稿人也在论文反驳阶段更积极参与，讨论时长更长。本工作证明，精心设计的LLM反馈可有效提升评审质量，使意见更具体、更具行动力，并增强审稿人与作者的互动。该系统已开源：https://github.com/zou-group/review_feedback_agent。
原文摘要 · Abstract (English)
Peer review at AI conferences is stressed by rapidly rising submission volumes, leading to deteriorating review quality and increased author dissatisfaction. To address these issues, we developed Review Feedback Agent, a system leveraging multiple large language models (LLMs) to improve review clarity and actionability by providing automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers. Implemented at ICLR 2025 as a large randomized control study, our system provided optional feedback to more than 20,000 randomly selected reviews. To ensure high-quality feedback for reviewers at this scale, we also developed a suite of automated reliability tests powered by LLMs that acted as guardrails to ensure feedback quality, with feedback only being sent to reviewers if it passed all the tests. The results show that 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers. This suggests that many reviewers found the AI-generated feedback sufficiently helpful to merit updating their reviews. Incorporating AI feedback led to significantly longer reviews (an average increase of 80 words among those who updated after receiving feedback) and more informative reviews, as evaluated by blinded researchers. Moreover, reviewers who were selected to receive AI feedback were also more engaged during paper rebuttals, as seen in longer author-reviewer discussions. This work demonstrates that carefully designed LLM-generated review feedback can enhance peer review quality by making reviews more specific and actionable while increasing engagement between reviewers and authors. The Review Feedback Agent is publicly available at https://github.com/zou-group/review_feedback_agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。