让大模型生成内容既易被检测又保持高任务性能。
LLM Output Detectability and Task Performance Can be Jointly Optimized
- 用DPO微调模型，同时优化检测率和任务表现。
- 在低误报率下检测效果媲美水印技术，任务性能更优。
- 仅需数千样本和少量算力，适配多种模型与攻击场景。
检测大模型生成文本对保障透明性与责任性至关重要。水印技术通过偏置词元分布，在输出中嵌入可检测信号，实现统计上可靠的检测。然而已有研究指出，水印化模型在下游任务上表现往往下降。本文提出PUPPET框架，通过直接偏好优化（DPO）微调大模型，使其生成的内容既更易被目标检测器识别，又在下游任务中表现更好。该方法引入两个奖励：一个输出机器生成概率的检测器，一个衡量特定任务指标的评估器。如同水印需密钥验证，此检测器专用设计使模型提供方可追踪其发布模型的实际使用情况。在长文本问答、摘要和作文生成任务上的实验表明，经PUPPET训练的模型在严格低误报率下仍具备与水印方法相当的检测能力，且在下游任务上表现更优。该优化仅需数千样本和1–2 GPU小时，其优势在跨领域任务、六种不同架构的检测器、多种模型家族与规模下均成立，且对改写攻击具有鲁棒性。
原文摘要 · Abstract (English)
Detecting machine-generated text is essential for transparency and accountability when deploying LLMs. Watermarking enables statistically reliable detection by biasing token distributions to embed detectable signals into LLM outputs. However, it has been reported that watermarked LLMs often perform worse on downstream tasks. We propose PUPPET, a framework that fine-tunes an LLM via DPO to generate text that is both more detectable by a target detector and better performing on downstream tasks. We use two rewards: a detector that outputs a machine-class likelihood and an evaluator that measures a task-specific metric. Just as a watermark is verified with its secret key, this detector-specific design lets an LLM provider track how its published model is used. Experiments on long-form QA, summarization, and essay writing show that LLMs trained with PUPPET achieve detectability competitive with watermarking methods---even at strict low FPRs---while outperforming them on downstream tasks. Moreover, this optimization requires only a few thousand samples and 1--2 GPU hours, and its gains hold across out-of-domain tasks, six detectors of diverse architectures, and different LLM families and sizes, and are even robust to paraphrasing attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。