用智能流程从大模型中复现训练过的版权内容
RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline
- 设计反馈循环，逐步修正生成内容以逼近原文
- 在30本书上测试，提取准确率提升24%
- 能突破模型拒绝响应的限制，适合研究数据泄露者
若无法查看大语言模型（LLM）的训练数据，我们如何知道它学到了什么？我们认为最有力的证据是模型能自主复现目标内容。为此，我们提出RECAP，一个用于从大模型输出中提取并验证记忆训练数据的智能流水线。其核心是一个反馈驱动的循环：初始生成结果由一个辅助语言模型评估，该模型将输出与参考文本比对并识别差异，再将最小化修正提示反馈给目标模型以引导后续生成。此外，为应对对齐机制导致的拒绝行为，RECAP还包含一个越狱模块，可检测并突破此类障碍。我们在新基准EchoTrace上进行评估，该基准涵盖30余本完整书籍。结果显示，相较于单次迭代方法，RECAP显著提升效果。例如，使用GPT-4.1时，版权文本提取的平均ROUGE-L得分从0.38提升至0.47，提高近24%。
原文摘要 · Abstract (English)
If we cannot inspect the training data of a large language model (LLM), how can we ever know what it has seen? We believe the most compelling evidence arises when the model itself freely reproduces the target content. As such, we propose RECAP, an agentic pipeline designed to elicit and verify memorized training data from LLM outputs. At the heart of RECAP is a feedback-driven loop, where an initial extraction attempt is evaluated by a secondary language model, which compares the output against a reference passage and identifies discrepancies. These are then translated into minimal correction hints, which are fed back into the target model to guide subsequent generations. In addition, to address alignment-induced refusals, RECAP includes a jailbreaking module that detects and overcomes such barriers. We evaluate RECAP on EchoTrace, a new benchmark spanning over 30 full books, and the results show that RECAP leads to substantial gains over single-iteration approaches. For instance, with GPT-4.1, the average ROUGE-L score for the copyrighted text extraction improved from 0.38 to 0.47 - a nearly 24% increase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。