arXiv:2509.04460cs.CLcs.AI2025-09ACL综述被引 2

区分AI生成内容与语言润色,提升学术评审检测准确性

CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection

  • 基于内容而非风格检测AI生成的评审文本
  • 覆盖六种人机协作模式,构建细粒度检测数据集
  • 适合关注学术诚信与AI工具合规性的研究者

大型语言模型(LLMs)在同行评审中的应用日益广泛,可能威胁学术评价的公平性与可靠性。尽管LLMs可辅助语言润色,但其生成实质性评审内容的风险日益凸显。现有通用AI生成文本检测方法易受改写攻击,难以区分表面语言优化与实质内容生成,主要依赖风格线索。应用于评审时,这会导致误判经允许的AI润色,却漏检伪装成人工的AI生成内容。为此,我们提出从风格导向转向内容导向的检测范式。具体地,构建了基于细粒度数据集的CoCoNUTS基准,涵盖六种人机协作模式的AI生成评审文本;并开发了基于多任务学习框架的CoCoDet检测器,实现对评审内容中AI参与的更准确、鲁棒的识别。本工作为评估LLMs在评审中的使用提供了实践基础,推动更精准、公平、可靠的现实应用检测方法发展。代码与数据将公开于https://github.com/Y1hanChen/COCONUTS。

原文摘要 · Abstract (English)

The growing integration of large language models (LLMs) into the peer review process presents potential risks to the fairness and reliability of scholarly evaluation. While LLMs offer valuable assistance for reviewers with language refinement, there is growing concern over their use to generate substantive review content. Existing general AI-generated text detectors are vulnerable to paraphrasing attacks and struggle to distinguish between surface language refinement and substantial content generation, suggesting that they primarily rely on stylistic cues. When applied to peer review, this limitation can result in unfairly suspecting reviews with permissible AI-assisted language enhancement, while failing to catch deceptively humanized AI-generated reviews. To address this, we propose a paradigm shift from style-based to content-based detection. Specifically, we introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews, covering six distinct modes of human-AI collaboration. Furthermore, we develop CoCoDet, an AI review detector via a multi-task learning framework, designed to achieve more accurate and robust detection of AI involvement in review content. Our work offers a practical foundation for evaluating the use of LLMs in peer review, and contributes to the development of more precise, equitable, and reliable detection methods for real-world scholarly applications. Our code and data will be publicly available at https://github.com/Y1hanChen/COCONUTS.

AI检测同行评审大模型风险

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。