arXiv:2412.10432cs.CLcs.AI2024-12AAAI被引 26

通过模仿机器写作风格,精准识别人类文本经机器修改的痕迹。

Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection

  • 先模仿机器风格分布,再对比待测文本与机器风格差异
  • 在六类大模型、三类修改类型上提升检测准确率13%~19%
  • 仅用1000样本和5分钟训练,超越商用检测器

大型语言模型(LLMs)革新了文本生成,使检测机器生成文本愈发困难。现有方法对纯机器生成文本检测效果良好,但对机器修订文本(重写、扩展、润色)的区分能力较差,因其与原始人类提示仅存在微小改动。由于内容可能源自人类输入,检测机器修订文本需识别机器特有的写作风格,如偏好用词。然而,现有方法难以捕捉隐藏于人类贡献内容中的机器风格。本文提出“先模仿后检测”(ImBD)方法:首先模仿机器风格的词元分布,再将待测文本与该分布对比以判断是否被机器修订。为此,引入风格偏好优化(SPO),使评分模型对齐机器生成文本的风格偏好。随后使用风格条件概率曲率(Style-CPC)量化原文本与条件采样文本间的对数概率差异,实现有效检测。我们在六种大模型、四个文本领域、三种机器修订类型下进行广泛实验。相比现有最优方法,本方法在开源模型修订文本检测上AUC提升13%,对GPT-3.5和GPT-4o修订文本分别提升5%和19%。尤为显著的是,仅用1000样本与5分钟SPO训练,即超越商业训练的GPT-Zero,验证了其高效性与有效性。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have revolutionized text generation, making detecting machine-generated text increasingly challenging. Although past methods have achieved good performance on detecting pure machine-generated text, those detectors have poor performance on distinguishing machine-revised text (rewriting, expansion, and polishing), which can have only minor changes from its original human prompt. As the content of text may originate from human prompts, detecting machine-revised text often involves identifying distinctive machine styles, e.g., worded favored by LLMs. However, existing methods struggle to detect machine-style phrasing hidden within the content contributed by humans. We propose the "Imitate Before Detect" (ImBD) approach, which first imitates the machine-style token distribution, and then compares the distribution of the text to be tested with the machine-style distribution to determine whether the text has been machine-revised. To this end, we introduce style preference optimization (SPO), which aligns a scoring LLM model to the preference of text styles generated by machines. The aligned scoring model is then used to calculate the style-conditional probability curvature (Style-CPC), quantifying the log probability difference between the original and conditionally sampled texts for effective detection. We conduct extensive comparisons across various scenarios, encompassing text revisions by six LLMs, four distinct text domains, and three machine revision types. Compared to existing state-of-the-art methods, our method yields a 13% increase in AUC for detecting text revised by open-source LLMs, and improves performance by 5% and 19% for detecting GPT-3.5 and GPT-4o revised text, respectively. Notably, our method surpasses the commercially trained GPT-Zero with just $1,000$ samples and five minutes of SPO, demonstrating its efficiency and effectiveness.

文本检测风格识别LLM安全机器修订

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。