提出锚定解码方法,降低大模型抄袭风险且不影响生成质量。
Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
- 通过锚定安全模型,动态控制生成内容与原始数据的距离。
- 在长文本任务中减少75%的可测量抄袭差距,保持接近原模型的流畅性。
- 适合需要版权合规的工业级应用,尤其适用于混合授权数据训练的模型。
语言模型容易记忆并复现训练数据中的片段,当源数据受版权保护时,可能引发创作者同意和补偿问题,以及开发者合规风险。本文提出锚定解码(Anchored Decoding),一种即插即用的推理阶段方法,可通过将生成过程约束在允许许可的安全模型附近,抑制原文复制行为。该方法自适应分配用户设定的信息预算,并在每步施加约束,实现序列级别的风险保障,支持风险与性能之间的可调平衡。为提升实用性,我们引入一个新训练的轻量级安全模型TinyComma 1.8B,以及基于ByteSampler框架的字节级变体Anchored$_{\mathrm{Byte}}$ Decoding,支持跨词汇融合。在六组模型对上的长文本评估显示,锚定解码及其字节级版本定义了新的帕累托前沿,在仅增加少量推理开销的情况下,将风险与安全参考之间的可测量抄袭差距缩小达75%,同时保留接近原始模型的流畅性和事实性。
原文摘要 · Abstract (English)
Language models (LMs) tend to memorize portions of their training data and emit verbatim spans. When the underlying sources are sensitive or copyright-protected, such reproduction raises issues of consent and compensation for creators and compliance risks for developers. We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying: it enables decoding from any risky LM trained on mixed-license data by keeping generation in bounded proximity to a permissively trained safe LM. Anchored Decoding adaptively allocates a user-chosen information budget over the generation trajectory and enforces per-step constraints that yield a sequence-level guarantee, enabling a tunable risk-utility trade-off. To make Anchored Decoding practically useful, we introduce a new permissively trained safe model (TinyComma 1.8B), as well as Anchored$_{\mathrm{Byte}}$ Decoding, a byte-level variant of our method that enables cross-vocabulary fusion via the ByteSampler framework (Hayase et al., 2025). Across six model pairs on long-form metrics for copying risk and utility, Anchored and Anchored$_{\mathrm{Byte}}$ Decoding define a new Pareto frontier, preserving near-original fluency and factuality while closing up to 75% of the measurable copying gap between the risky baseline and a safe reference, at a modest inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。