让大模型学会按指令嵌入可检测水印,且不影响回答质量。
Learning to Follow In-Context Watermark Instructions via Self-Distillation
- 用自蒸馏加逻辑扰动训练模型识别并响应水印指令。
- 在三个水印指令族上,检测率从0.100提升至0.974。
- 无需强模型指导或人工标注,适合需可控水印的部署场景。
上下文水印(ICW)在查询前添加指令,要求模型在其回复中嵌入统计可检测的信号,从而为大模型提供第三方可调用的水印接口,无需访问模型内部。其可靠性依赖于模型准确执行指令且不降低回答质量,但当前大模型的表现尚未被量化评估。为此,本文提出 $ ext{ICWBench}$,一个包含三个可验证的 ICW 指令族的基准,每项均以可检测性与回答质量双重评分。评估了14个前沿的专有及开源大模型,发现无一能在全部三类指令上同时达成高可检测性与高质量输出。为解决此问题,我们提出一种自包含的两阶段训练方法:第一阶段采用带有逻辑值扰动的自蒸馏(SDLP),使用同一基础模型作为教师和学生;通过解码时的指令等效逻辑扰动使教师遵循水印指令,学生则学习匹配教师输出分布。第二阶段引入强化学习,以自动验证器作为奖励函数。该方法应用于 Qwen3-14B 与 GPT-OSS-20B,使三种 ICW 指令的平均 TPR@1%FPR 分别从 0.100 提升至 0.974,从 0.337 提升至 0.968,同时在困惑度和大模型评价值上保持高回答质量。
原文摘要 · Abstract (English)
In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response. It thus equips LLMs with a watermarking interface that third parties can invoke without access to model internals. Its reliability hinges on the LLM following the instruction without degrading answer quality, yet how well current LLMs do so has not been measured. We introduce $\mathsf{ICWBench}$, a benchmark of three verifiable ICW instruction families, each scored on both detectability and answer quality. Evaluating 14 frontier proprietary and open-source LLMs, we find that none of the evaluated LLMs achieves both objectives across all three families. To address this, we propose a self-contained two-stage training method, requiring no distillation from a stronger model, no manual annotation, and no pre-existing ICW IF ability. The first stage, self-distillation with logits perturbation (SDLP), uses the same base LLM as both teacher and student: an instruction-equivalent decoding-time logits perturbation makes the teacher follow the ICW instruction, and the student is trained to match the teacher's output distribution. The second stage applies reinforcement learning with the automatic verifier as the reward. Applied to Qwen3-14B and GPT-OSS-20B, our method raises average TPR@$1\%$FPR across three ICW instructions from $0.100$ to $0.974$ and from $0.337$ to $0.968$, respectively, while maintaining high response quality under both perplexity evaluation and LLM-as-a-Judge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。