通过分析评审观点差异，区分人工与AI生成的论文评审。
Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews
- 结合文本特征与观点级语义分析，对比目标评审与多份AI生成评审。
- 在2万份评审数据中，误报率0.1%时准确率提升25.5%。
- 能有效识别经大模型润色的人工评审，误判率低于3.5%。
如何区分一篇同行评审是人工撰写还是由AI模型生成？我们认为，作者身份不应仅基于文本特征判断，还应考察其表达的思想、判断和主张。为此，我们提出Sem-Detect，一种结合文本特征与观点级语义分析的同行评审作者归属检测方法。该方法将目标评审与同一论文的多份AI生成评审进行对比，利用不同AI模型倾向于产生相似观点、而人类评审更具独特性和多样性的观察。实验结果表明，在包含ICLR和NeurIPS会议超过20,000份评审的数据集上，Sem-Detect在二分类任务中相较最强基线，于0.1%误报率下提升了25.5%的真正率。在三分类场景中，我们发现经大模型润色的人工评审仍保留人类语义特征，显著区别于纯AI生成文本；仅有少于3.5%的此类评审被错误归类为AI生成。
原文摘要 · Abstract (English)
How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in [email protected]% FPR in the binary setting. Moreover, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。