提出新方法评估大模型近似复现风险，效率提升万倍
Estimating near-verbatim extraction risk in language models with decoding-constrained beam search
- 用约束束搜索替代蒙特卡洛采样，快速估算近似复现风险
- 每条序列仅需约20次采样成本，提取风险比原文法高数倍
- 可发现原方法忽略的隐私与版权隐患，适合模型安全研究者
现有贪婪解码提取方法无法捕捉生成序列间提取风险的差异。概率提取虽能改进，但仅适用于完全复现情况，难以覆盖具有相似隐私与版权风险的近似复现实例。量化近似复现风险代价高昂：近似复现后缀集合呈组合爆炸，可靠蒙特卡洛估计需每序列约10万次采样。为此，我们提出解码约束束搜索，以确定性下界形式在相当于约20次蒙特卡洛采样的成本下，实现近似复现风险的高效估算。实验表明，该方法揭示了原文法无法发现的信息：更多可提取序列、显著更高的单序列提取质量，并展现出不同模型规模与文本类型下的风险分布模式。
原文摘要 · Abstract (English)
Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences. Probabilistic extraction -- computing the probability of generating a target suffix given a prefix under a decoding scheme -- addresses this, but is tractable only for verbatim memorization, missing near-verbatim instances that pose similar privacy and copyright risks. Quantifying near-verbatim extraction risk is expensive: the set of near-verbatim suffixes is combinatorially large, and reliable Monte Carlo (MC) estimation can require ~100,000 samples per sequence. To mitigate this cost, we introduce decoding-constrained beam search, which yields deterministic lower bounds on near-verbatim extraction risk at a cost comparable to ~20 MC samples per sequence. Across experiments, our approach surfaces information invisible to verbatim methods: many more extractable sequences, substantially larger per-sequence extraction mass, and patterns in how near-verbatim extraction risk manifests across model sizes and types of text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。