研究强化学习对齐如何影响大模型文本质量与可检测性
Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts
- 用人类反馈强化学习优化文本生成质量
- 优化后文本更长、重复，反而更容易被检测
- 零样本检测器比训练型更抗干扰，适合实际应用
大型语言模型在多项下游NLP任务中表现出色，生成的文本高度接近人类写作。然而，这种高相似性引发了恶意使用风险，因为模型生成文本越来越难以与真人写作区分。尽管已有检测方法，但攻击者仍可进一步编辑文本以降低可检测性。本文研究了通过人类反馈强化学习（RLHF）进一步优化文本后，对两类任务的生成质量及检测性能的影响：(a) 生成文本质量；(b) 基于训练和零样本的检测器表现。结果显示，虽然RLHF提升了文本质量，但也导致输出更长、更重复，且更易被检测。此外，基于训练的检测器在短文本和含代码文本上表现脆弱，而零样本检测器展现出更强鲁棒性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated exceptional performance on a range of downstream NLP tasks by generating text that closely resembles human writing. However, the ease of achieving this similarity raises concerns from potential malicious uses at scale by bad actors, as LLM-generated text becomes increasingly difficult to discern from human text. Although detection methods have been developed to address this issue, bad actors can further manipulate LLM-generated texts to make them less detectable. In this work, we study how further editing texts with Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs with human preferences, affects (a) the quality of generated texts for two tasks, and (b) the performance of LLM-generated text detectors, looking at both training-based and zero-shot detection methods. Although RLHF improves the quality of LLM-generated texts, we find that it also tends to produce more detectable, lengthy, and repetitive outputs. Additionally, we observe that training-based detectors are vulnerable to short texts and to texts that incorporate code, whereas zero-shot detectors exhibit greater robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。