检测论文评审中是否存在AI生成内容，发现现有方法效果不佳。
Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review
- 对比人类与GPT-4o撰写的评审文本，测试现有检测工具效果。
- 现有方法对GPT-4o评审识别率低，且误报率高。
- 提出新检测方法，在低误报下更好识别AI生成评审。
同行评审是保障科研成果可信性的关键环节，其公信力依赖于领域专家对投稿论文价值的审慎评估。随着大语言模型（LLMs）语言能力迅速提升，一种新风险浮现：不负责的审稿人可能利用LLM代为完成耗时的评审工作。本研究考察现有AI文本检测算法区分人工撰写与先进LLM（如GPT-4o）生成评审的能力。结果表明，现有方法难以有效识别大多数GPT-4o撰写的评审，同时伴随高误报率。为此，我们提出一种新检测方法，在低误报水平下显著优于现有技术。研究揭示了在个体评审层面准确识别AI生成内容的困难，凸显了亟需开发更有效的工具以应对生成式AI在学术评审中的不当应用。
原文摘要 · Abstract (English)
Peer review is a critical process for ensuring the integrity of published scientific research. Confidence in this process is predicated on the assumption that experts in the relevant domain give careful consideration to the merits of manuscripts which are submitted for publication. With the recent rapid advancements in the linguistic capabilities of large language models (LLMs), a new potential risk to the peer review process is that negligent reviewers will rely on LLMs to perform the often time consuming process of reviewing a paper. In this study, we investigate the ability of existing AI text detection algorithms to distinguish between peer reviews written by humans and different state-of-the-art LLMs. Our analysis shows that existing approaches fail to identify many GPT-4o written reviews without also producing a high number of false positive classifications. To address this deficiency, we propose a new detection approach which surpasses existing methods in the identification of GPT-4o written peer reviews at low levels of false positive classifications. Our work reveals the difficulty of accurately identifying AI-generated text at the individual review level, highlighting the urgent need for new tools and methods to detect this type of unethical application of generative AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。