让大模型无需检索就能记住文档知识,三阶段训练提升问答准确率。
Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization

- 分三步训练:注入文档、对齐问答、恢复通用能力
- 在8个场景中7个超越基线,领域问答准确率平均提升3.6个百分点
- 适合需要离线使用文档知识的模型部署场景
大型语言模型在推理时无法访问源文档时,难以回答关于限定文档集的问题。本文研究文档知识内化任务:将固定语料库转化为可直接使用的参数化知识,实现无检索问答。提出IAR(Inject, Align, Recover)三阶段后训练框架,分别完成结构化文档知识注入、问答行为对齐和通用能力恢复。不同于传统继续预训练,Inject将源文档转为续写、重写和指令条件重建目标;Align使用仅答案的问答监督微调注入模型;Recover将领域适配模型与基础指令模型融合,恢复通用能力。在Common Corpus(CC)和CCI数据集上,覆盖Llama、Phi、Qwen和SmolLM模型家族,IAR显著提升检索无关文档内化的性能边界。主实验中,IAR在8个数据集-模型组合中的7个超过原始SFT,在7项指标上表现更优,领域问答准确率平均提升3.6个百分点,IFEval、MMLU和MSBench三项通用评估平均提升12.1个百分点。扩展基线对比显示,LoRA和FAPM虽能提升个别通用指标,但在保持领先或接近领先的文档内化效果下,IAR仍具最强综合表现。
原文摘要 · Abstract (English)
Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。