水印会破坏大模型的安全性与有用性,新方法通过重采样恢复对齐性能。
Watermarking Degrades Alignment in Language Models: Analysis and Mitigation
- 提出在生成时多次采样并选最对齐结果的重采样策略
- 仅需2-4次采样即可基本恢复未加水印的对齐表现
- 首次实证揭示水印导致安全与有用性失衡,适合模型部署者参考
水印已成为追踪语言模型输出的有效手段,但在推理时修改了原本经过对齐训练优化的词元概率分布。这种调整与模型安全性和有用性设计初衷产生冲突。对多个主流模型及两种典型水印方案的实验表明,水印引入了非平凡、有规律但模型特异的对齐偏移。观察到两种失效模式:防护能力减弱(更乐于助人但更不安全)和防护过度强化(拒绝响应过于保守)。这些影响即使在控制困惑度下降后仍存在,说明是专属对齐的扭曲而非单纯质量下降。为此提出对齐重采样(AR)方法:通过外部奖励模型从多个水印输出中筛选最对齐的结果。基于高斯随机变量最大值的期望理论,推导出对齐提升随样本量呈亚对数增长的下界。实践中,仅需2至4个候选样本,即能显著恢复真实性、安全性和有用性上的未水印水平,且不影响水印检测。这是首个关于水印与对齐交互的实证研究,表明简单的推理阶段修复可有效恢复对齐。
原文摘要 · Abstract (English)
Watermarking has become a practical tool for tracing language model outputs, but it modifies token probabilities at inference time, which were carefully tuned by alignment training. This creates a tension: how do watermark-induced shifts interact with the procedures intended to make models safe and useful? Experiments on several contemporary models and two representative watermarking schemes reveal that watermarking induces a nontrivial, patterned yet model-specific shift in alignment. We see two failure modes: guard attenuation, where models become more helpful but less safe, and guard amplification, where refusals become overly conservative. These effects persist even after controlling for perplexity degradation, pointing to alignment-specific distortions, not just quality loss. We address this with Alignment Resampling (AR), a procedure that samples multiple watermarked outputs and selects the most aligned response according to an external reward model. Using standard results on the expected maximum of Gaussian random variables, we derive a theoretical lower bound showing that alignment gains grow sublogarithmically with sample size. In practice, sampling as few as two to four candidates largely restores unwatermarked alignment performance in truthfulness, safety, and helpfulness, without hurting watermark detection. This is the first empirical study of watermarking-alignment interactions; it shows that a simple inference-time fix can recover alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。