用深度上下文蒸馏训练轻量知识模块，实现高效文档知识注入。
Training Plug-n-Play Knowledge Modules with Deep Context Distillation
- 将文档知识封装为可插拔的轻量LoRA模块，按需注入模型。
- 通过模拟教师模型隐藏状态和输出，提升知识模块效果。
- 适合低数据、私有文档场景，与RAG协同效果更优。
大语言模型预训练后动态集成新信息仍具挑战，尤其在数据稀缺或处理私有专有文档时。上下文学习和检索增强生成（RAG）存在推理成本高、无法捕捉全局文档信息等局限。本文提出通过训练文档级知识模块（KMs）实现知识模块化。KMs为轻量级参数高效LoRA模块，可按需插入模型，存储新文档信息。我们发现，以下一个词预测作为训练目标对KMs效果不佳，因此提出深度上下文蒸馏：通过学习使KMs参数能模拟教师模型在文档上下文中的隐藏状态与输出逻辑。该方法在两个数据集上均优于标准的下一个词预测和预指令训练。最后，我们揭示了KMs与RAG之间的协同潜力。
原文摘要 · Abstract (English)
Dynamically integrating new or rapidly evolving information after (Large) Language Model pre-training remains challenging, particularly in low-data scenarios or when dealing with private and specialized documents. In-context learning and retrieval-augmented generation (RAG) face limitations, including their high inference costs and their inability to capture global document information. In this paper, we propose a way of modularizing knowledge by training document-level Knowledge Modules (KMs). KMs are lightweight components implemented as parameter-efficient LoRA modules, which are trained to store information about new documents and can be easily plugged into models on demand. We show that next-token prediction performs poorly as the training objective for KMs. We instead propose Deep Context Distillation: we learn KMs parameters such as to simulate hidden states and logits of a teacher that takes the document in context. Our method outperforms standard next-token prediction and pre-instruction training techniques, across two datasets. Finally, we highlight synergies between KMs and RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。