用大模型生成假评审数据，提升识别不合格审稿的准确率
ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation
- 通过GPT-4生成6634篇论文的合成评审，扩充真实数据集
- 混合训练后，模型召回率达0.6653，F1达0.7073
- 适合关注学术诚信与AI治理的研究者和期刊编辑
同行评审是科学的守门人，但投稿量激增及大语言模型在学术评价中的广泛应用带来了前所未有的挑战。尽管近期研究聚焦于利用大模型提升评审效率，但人类与AI系统产生的不合格评审若不加控制，将系统性损害学术诚信。为此，我们提出ReviewGuard，一个基于四阶段大模型驱动框架的自动化系统：从ICLR和NeurIPS的OpenReview平台收集数据，使用GPT-4.1标注并经人工验证，生成6,634篇论文的24,657条真实评审和46,438条合成评审，再对编码器模型与开源大模型进行微调。特征分析显示，不合格评审评分更低、自报信心更高、结构更简单、情感更负面。AI文本检测表明，自ChatGPT发布以来，AI撰写的评审比例显著上升。混合训练真实与合成数据显著提升检测性能——例如，Qwen 3-8B的召回率从0.5499提升至0.6653，F1从0.5606提升至0.7073。本研究首次构建了基于大模型的不合格评审检测系统，为学术评审中的人工智能治理提供实证支持。代码、提示词与数据已公开于https://github.com/haoxuan-unt2024/ReviewGuard。
原文摘要 · Abstract (English)
Peer review serves as the gatekeeper of science, yet the surge in submissions and widespread adoption of large language models (LLMs) in scholarly evaluation present unprecedented challenges. While recent work has focused on using LLMs to improve review efficiency, unchecked deficient reviews from both human experts and AI systems threaten to systematically undermine academic integrity. To address this issue, we introduce ReviewGuard, an automated system for detecting and categorizing deficient reviews through a four-stage LLM-driven framework: data collection from ICLR and NeurIPS on OpenReview, GPT-4.1 annotation with human validation, synthetic data augmentation yielding 6,634 papers with 24,657 real and 46,438 synthetic reviews, and fine-tuning of encoder-based models and open-source LLMs. Feature analysis reveals that deficient reviews exhibit lower rating scores, higher self-reported confidence, reduced structural complexity, and more negative sentiment than sufficient reviews. AI-generated text detection shows dramatic increases in AI-authored reviews since ChatGPT's emergence. Mixed training with synthetic and real data substantially improves detection performance - for example, Qwen 3-8B achieves recall of 0.6653 and F1 of 0.7073, up from 0.5499 and 0.5606 respectively. This study presents the first LLM-driven system for detecting deficient peer reviews, providing evidence to inform AI governance in peer review. Code, prompts, and data are available at https://github.com/haoxuan-unt2024/ReviewGuard
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。