让大模型在不查资料的情况下也能准确引用训练时见过的文档。
Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
- 通过持续预训练绑定事实与文档标识，实现无检索引用。
- 引入主动索引技术，使模型在改写和组合中仍能正确溯源。
- 实测精度提升最高达30.2%，适合追求可信生成的开发者使用。
可信的语言模型应提供准确且可验证的答案。当前系统依赖外部检索器在推理时添加引用，带来延迟、架构依赖和检索噪声问题。本文探索是否可通过改进训练过程，使大模型在不依赖测试时检索的前提下，可靠地对持续预训练中接触过的文档进行引用。为此，构建了包含真实语料（Wikipedia、Common Crawl、arXiv）与新文档的CitePretrainBench基准，涵盖单事实与多事实引用任务。方法分为两阶段：（1）持续预训练，通过绑定事实与持久文档标识来索引知识；（2）指令微调以激发引用行为。提出主动索引机制，通过合成数据增强训练，包括多样化重述事实及双向训练（源→事实、事实→源），提升模型对改写与组合的鲁棒性。Qwen-2.5-7B&3B实验表明，主动索引相比被动索引基准（仅在文档后附加标识符）在所有任务和模型上平均提升30.2%的引用精度。消融实验显示，随着增强数据量增加，性能持续上升，甚至在原始数据量16倍时仍呈增长趋势。最终证明，内部引用可弥补外部检索缺陷，增强对检索噪声的鲁棒性。
原文摘要 · Abstract (English)
Trustworthy language models should provide both correct and verifiable answers. However, citations generated directly by standalone LLMs are often unreliable. As a result, current systems insert citations by querying an external retriever at inference time, introducing latency, infrastructure dependence, and vulnerability to retrieval noise. We explore whether LLMs can be made to reliably attribute to the documents seen during continual pretraining without test-time retrieval, by revising the training process. To study this, we construct CitePretrainBench, a benchmark that mixes real-world corpora (Wikipedia, Common Crawl, arXiv) with novel documents and probes both short-form (single-fact) and long-form (multi-fact) citation tasks. Our approach follows a two-stage process: (1) continual pretraining to index factual knowledge by binding it to persistent document identifiers; and (2) instruction tuning to elicit citation behavior. We introduce Active Indexing for the first stage, which creates generalizable, source-anchored bindings by augmenting training with synthetic data that (i) restate each fact in diverse, compositional forms and (ii) enforce bidirectional training (source-to-fact and fact-to-source). This equips the model to both generate content from a cited source and attribute its own answers, improving robustness to paraphrase and composition. Experiments with Qwen-2.5-7B&3B show that Active Indexing consistently outperforms a Passive Indexing baseline, which simply appends an identifier to each document, achieving citation precision gains of up to 30.2% across all tasks and models. Our ablation studies reveal that performance continues to improve as we scale the amount of augmented data, showing a clear upward trend even at 16x the original token count. Finally, we show that internal citations complement external ones by making the model more robust to retrieval noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。