通过分析情感分布稳定性,提升大模型生成文本的检测鲁棒性。
Model-Agnostic Sentiment Distribution Stability Analysis for Robust LLM-Generated Texts Detection
- 利用情感分布一致性与保留性双指标,不依赖具体模型检测生成文本。
- 在多个大模型上实现超过16%的F1分数提升,显著优于现有方法。
- 对改写、对抗攻击等挑战场景表现稳健,适合实际应用部署。
大语言模型(LLMs)生成内容日益复杂,传统检测方法因泛化能力差、易受改写和对抗扰动影响而失效。本文提出SentiDetect,一种基于情感分布稳定性的模型无关检测框架。核心思路是:大模型输出情感模式更一致,人类文本则情感波动更大。我们定义两个互补指标——情感分布一致性与情感分布保留性,分别衡量在情感改变和语义保持变换下的稳定性。在五个数据集及Gemini-1.5-Pro、Claude-3、GPT-4-0613、LLaMa-3.3等多模型上测试,结果表明,SentiDetect在Gemini-1.5-Pro和GPT-4-0613上分别取得超过16%和11%的F1分数提升。同时,在改写、对抗攻击和长度变化等挑战场景中也表现出更强鲁棒性,优于当前最优基线。
原文摘要 · Abstract (English)
The rapid advancement of large language models (LLMs) has resulted in increasingly sophisticated AI-generated content, posing significant challenges in distinguishing LLM-generated text from human-written language. Existing detection methods, primarily based on lexical heuristics or fine-tuned classifiers, often suffer from limited generalizability and are vulnerable to paraphrasing, adversarial perturbations, and cross-domain shifts. In this work, we propose SentiDetect, a model-agnostic framework for detecting LLM-generated text by analyzing the divergence in sentiment distribution stability. Our method is motivated by the empirical observation that LLM outputs tend to exhibit emotionally consistent patterns, whereas human-written texts display greater emotional variability. To capture this phenomenon, we define two complementary metrics: sentiment distribution consistency and sentiment distribution preservation, which quantify stability under sentiment-altering and semantic-preserving transformations. We evaluate SentiDetect on five diverse datasets and a range of advanced LLMs,including Gemini-1.5-Pro, Claude-3, GPT-4-0613, and LLaMa-3.3. Experimental results demonstrate its superiority over state-of-the-art baselines, with over 16% and 11% F1 score improvements on Gemini-1.5-Pro and GPT-4-0613, respectively. Moreover, SentiDetect also shows greater robustness to paraphrasing, adversarial attacks, and text length variations, outperforming existing detectors in challenging scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。