用自蒸馏方法让大模型高效吸收新知识，效果优于传统微调和RAG。
Efficient Knowledge Injection in LLMs via Self-Distillation
- 用提示自蒸馏技术从自由文本中内化新事实知识。
- 在多个模型规模和架构上均超越监督微调，接近甚至超过RAG性能。
- 无需更大教师模型或结构化知识，适合实际部署场景。
在许多实际应用中，大型语言模型（LLMs）需要获取其预训练数据中未包含的新知识。高效利用这些知识通常依赖于监督微调或检索增强生成（RAG）。尽管RAG已成为知识注入的行业标准，但微调尚未取得同等成功。本文提出利用提示蒸馏——一种此前主要用于风格对齐和指令微调的自蒸馏方法——从自由格式文档中内化新的事实性知识。与先前方法不同，本方法无需更大教师模型，也无需结构化知识格式。在多个模型规模和模型家族中，我们证明提示蒸馏优于标准监督微调，甚至可超越RAG。我们分析了促使提示蒸馏有效性的关键因素，并考察其扩展特性。
原文摘要 · Abstract (English)
In many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG). Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillation-based method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation's effectiveness and examine how it scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。