arXiv:2410.16659cs.CLcs.AI2024-10被引 4

精准定位机器生成文本中的每处机器痕迹,突破现有方法局限。

RKadiyala at SemEval-2024 Task 8: Black-Box Word-Level Text Boundary Detection in Partially Machine Generated Texts

  • 基于词级边界检测,识别文本中机器生成部分。
  • 在未见领域和生成器上表现显著优于现有系统。
  • 适用于指令微调大模型输出的检测,实用性强。

随着生成式模型在文本生成中的广泛应用,区分人类写作与机器生成文本成为关键挑战。现有模型和专有系统多聚焦于判断文本整体是否为机器生成,少数系统可在句或段级别提供生成概率,但准确率低,仅适用于特定领域和生成器。本文提出若干可靠方法,解决全新的词级文本边界检测任务,对比不同方法效果,并评估模型在未见领域和生成器上的性能。结果表明,检测准确率显著提升,且在多种检测能力维度表现更优。同时讨论了未来改进方向及研究意义。所提模型特别适用于检测众多LLM指令变体输出中的机器生成部分。

原文摘要 · Abstract (English)

With increasing usage of generative models for text generation and widespread use of machine generated texts in various domains, being able to distinguish between human written and machine generated texts is a significant challenge. While existing models and proprietary systems focus on identifying whether given text is entirely human written or entirely machine generated, only a few systems provide insights at sentence or paragraph level at likelihood of being machine generated at a non reliable accuracy level, working well only for a set of domains and generators. This paper introduces few reliable approaches for the novel task of identifying which part of a given text is machine generated at a word level while comparing results from different approaches and methods. We present a comparison with proprietary systems , performance of our model on unseen domains' and generators' texts. The findings reveal significant improvements in detection accuracy along with comparison on other aspects of detection capabilities. Finally we discuss potential avenues for improvement and implications of our work. The proposed model is also well suited for detecting which parts of a text are machine generated in outputs of Instruct variants of many LLMs.

文本检测词级定位大模型安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。