提出无需原始模型的低熵文本水印方法,兼顾安全与效率。
Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking
- 用轻量特征提取器预测下一个词的熵值高低
- 自适应阈值导航使水印比例随绿色词增加而下降
- 99%参数量减少,性能媲美顶尖方法
基于logit的LLM水印通过维护绿色和红色词表,在生成时提高绿色词概率来追踪和验证AI生成内容。然而在低熵场景下,由于输出可预测,难以选择绿色词而不破坏自然文本流。现有方法依赖原始LLM计算熵,仅对高熵词进行水印,但存在两大问题:(1) 高计算成本和检测延迟;(2) 模型泄露风险。为此,我们提出Invisible Entropy(IE),一种提升安全性和效率的水印范式。IE不依赖原始模型,引入轻量特征提取器与熵标签器,预测下一词熵值高低,并基于理论分析设计自适应阈值导航器,确定一个水印率随绿色词数量上升而下降的阈值,提升水印文本自然性并增强检测鲁棒性。在HumanEval和MBPP数据集上的实验表明,IE实现99%参数量压缩,性能达到当前最优水平。本工作为低熵水印提供了安全高效的解决方案。
原文摘要 · Abstract (English)
Logit-based LLM watermarking traces and verifies AI-generated content by maintaining green and red token lists and increasing the likelihood of green tokens during generation. However, it fails in low-entropy scenarios, where predictable outputs make green token selection difficult without disrupting natural text flow. Existing approaches address this by assuming access to the original LLM to calculate entropy and selectively watermark high-entropy tokens. However, these methods face two major challenges: (1) high computational costs and detection delays due to reliance on the original LLM, and (2) potential risks of model leakage. To address these limitations, we propose Invisible Entropy (IE), a watermarking paradigm designed to enhance both safety and efficiency. Instead of relying on the original LLM, IE introduces a lightweight feature extractor and an entropy tagger to predict whether the entropy of the next token is high or low. Furthermore, based on theoretical analysis, we develop a threshold navigator that adaptively sets entropy thresholds. It identifies a threshold where the watermark ratio decreases as the green token count increases, enhancing the naturalness of the watermarked text and improving detection robustness. Experiments on HumanEval and MBPP datasets demonstrate that IE reduces parameter size by 99\% while achieving performance on par with state-of-the-art methods. Our work introduces a safe and efficient paradigm for low-entropy watermarking. https://github.com/Carol-gutianle/IE https://huggingface.co/datasets/Carol0110/IE-Tagger
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。