构建可提升检测器泛化能力的机器生成文本基准,增强文本类人对齐性。
MAGA-Bench: Machine-Augment-Generated Text via Alignment Detection Benchmark
- 通过提示工程与生成-检测对抗强化学习,提升生成文本的人类对齐度。
- 基于MAGA训练的检测器泛化AUC平均提升4.60%,而原有检测器性能下降8.13%。
- 适合关注生成文本检测泛化能力与对抗性评估的研究者使用。
机器生成文本(MGT)正变得越来越难以与人类写作文本(HWT)区分,加剧了虚假新闻和网络欺诈等恶意行为。现有检测器的泛化能力严重依赖数据集质量,单纯扩展生成来源已难以为继,需进一步增强生成过程。基于HC-Var理论,提升MGT的人类对齐性不仅能有效测试检测器鲁棒性,还能增强在对齐数据上微调的检测器泛化能力。为此,我们提出机器-增强-生成文本通过对齐(MAGA)检测基准。MAGA整合多种对齐方法,涵盖提示构造、生成-检测对抗强化学习(GDARL)及推理过程。实验表明,基于MAGA微调的RoBERTa检测器在泛化AUC上平均提升4.60%;而所生成的对齐文本使部分检测器的AUC平均下降8.13%。我们希望MAGA基准能为未来机器生成文本检测器泛化能力研究提供重要参考。
原文摘要 · Abstract (English)
Machine-Generated Text (MGT) is becoming increasingly difficult to distinguish from Human-Written Text (HWT). This trend has exacerbated malicious activities such as fake news and online fraud. The generalization ability of fine-tuned detectors relies heavily on dataset quality, and simply expanding the sources of MGT may become increasingly insufficient. Further augmentation of the generation process is required. Based on HC-Var's theory, enhancing the human-like alignment of MGT not only facilitates robustness testing of existing detectors but also boosts the generalization ability of detectors fine-tuned on such aligned MGT datasets. Therefore, we propose the \textbf{M}achine-\textbf{A}ugment-\textbf{G}enerated Text via \textbf{A}lignment (MAGA) Detection Benchmark. MAGA integrates several alignment methods, ranging from prompt construction to \textbf{G}enerator-\textbf{D}etector \textbf{A}dversarial \textbf{R}einforcement \textbf{L}earning (GDARL) and the reasoning process. In our experiments, the RoBERTa detector fine-tuned on MAGA achieves an average improvement of 4.60\% in generalization AUC. Conversely, the aligned MGTs in MAGA also lead to an average decrease of 8.13\% in the AUC of selected detectors. We hope the MAGA Benchmark will provide valuable insights for future research on the generalization ability of MGT detectors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。